CLR Explained
-
Upload
akshay-mehta -
Category
Documents
-
view
90 -
download
1
description
Transcript of CLR Explained
Introduction to Algorithms6.046J/18.401J
Prof. Charles E. Leiserson
LECTURE 1Analysis of Algorithms• Insertion sort• Merge sort
September 8, 2004 Introduction to Algorithms L1.2
Course information
1. Staff2. Prerequisites3. Lectures4. Recitations5. Handouts6. Textbook (CLRS)
7. Extra help8. Registration 9.Problem sets10.Describing algorithms11.Grading policy12.Collaboration policy
Course information handout
© 2001–4 by Charles E. Leiserson
September 8, 2004 Introduction to Algorithms L1.3
Analysis of algorithms
The theoretical study of computer-program performance and resource usage.
What’s more important than performance?• modularity• correctness• maintainability• functionality• robustness
• user-friendliness• programmer time• simplicity• extensibility• reliability
© 2001–4 by Charles E. Leiserson
September 8, 2004 Introduction to Algorithms L1.4
Why study algorithms and performance?
• Algorithms help us to understand scalability.• Performance often draws the line between what
is feasible and what is impossible.• Algorithmic mathematics provides a language
for talking about program behavior.• Performance is the currency of computing.• The lessons of program performance generalize
to other computing resources. • Speed is fun!
© 2001–4 by Charles E. Leiserson
September 8, 2004 Introduction to Algorithms L1.5
The problem of sorting
Input: sequence ⟨a1, a2, …, an⟩ of numbers.
Example:Input: 8 2 4 9 3 6
Output: 2 3 4 6 8 9
Output: permutation ⟨a'1, a'2, …, a'n⟩ suchthat a'1 ≤ a'2 ≤ … ≤ a'n .
© 2001–4 by Charles E. Leiserson
September 8, 2004 Introduction to Algorithms L1.6
Insertion sortINSERTION-SORT (A, n) ⊳ A[1 . . n]
for j ← 2 to ndo key ← A[ j]
i ← j – 1while i > 0 and A[i] > key
do A[i+1] ← A[i]i ← i – 1
A[i+1] = key
“pseudocode”
© 2001–4 by Charles E. Leiserson
September 8, 2004 Introduction to Algorithms L1.7
Insertion sortINSERTION-SORT (A, n) ⊳ A[1 . . n]
for j ← 2 to ndo key ← A[ j]
i ← j – 1while i > 0 and A[i] > key
do A[i+1] ← A[i]i ← i – 1
A[i+1] = key
“pseudocode”
sorted
i j
keyA:
1 n
© 2001–4 by Charles E. Leiserson
September 8, 2004 Introduction to Algorithms L1.8
Example of insertion sort8 2 4 9 3 6
© 2001–4 by Charles E. Leiserson
September 8, 2004 Introduction to Algorithms L1.9
Example of insertion sort8 2 4 9 3 6
© 2001–4 by Charles E. Leiserson
September 8, 2004 Introduction to Algorithms L1.8
Example of insertion sort8 2 4 9 3 6
© 2001–4 by Charles E. Leiserson
September 8, 2004 Introduction to Algorithms L1.11
Example of insertion sort8 2 4 9 3 6
2 8 4 9 3 6
© 2001–4 by Charles E. Leiserson
September 8, 2004 Introduction to Algorithms L1.12
Example of insertion sort8 2 4 9 3 6
2 8 4 9 3 6
2 4 8 9 3 6
© 2001–4 by Charles E. Leiserson
September 8, 2004 Introduction to Algorithms L1.13
Example of insertion sort8 2 4 9 3 6
2 8 4 9 3 6
2 4 8 9 3 6
© 2001–4 by Charles E. Leiserson
September 8, 2004 Introduction to Algorithms L1.14
Example of insertion sort8 2 4 9 3 6
2 8 4 9 3 6
2 4 8 9 3 6
2 4 8 9 3 6
© 2001–4 by Charles E. Leiserson
September 8, 2004 Introduction to Algorithms L1.15
Example of insertion sort8 2 4 9 3 6
2 8 4 9 3 6
2 4 8 9 3 6
2 4 8 9 3 6
© 2001–4 by Charles E. Leiserson
September 8, 2004 Introduction to Algorithms L1.16
Example of insertion sort8 2 4 9 3 6
2 8 4 9 3 6
2 4 8 9 3 6
2 4 8 9 3 6
2 3 4 8 9 6
© 2001–4 by Charles E. Leiserson
September 8, 2004 Introduction to Algorithms L1.17
Example of insertion sort8 2 4 9 3 6
2 8 4 9 3 6
2 4 8 9 3 6
2 4 8 9 3 6
2 3 4 8 9 6
© 2001–4 by Charles E. Leiserson
September 8, 2004 Introduction to Algorithms L1.18
Example of insertion sort8 2 4 9 3 6
2 8 4 9 3 6
2 4 8 9 3 6
2 4 8 9 3 6
2 3 4 8 9 6
2 3 4 6 8 9 done© 2001–4 by Charles E. Leiserson
September 8, 2004 Introduction to Algorithms L1.19
Running time
• The running time depends on the input: an already sorted sequence is easier to sort.
• Parameterize the running time by the size of the input, since short sequences are easier to sort than long ones.
• Generally, we seek upper bounds on the running time, because everybody likes a guarantee.
© 2001–4 by Charles E. Leiserson
September 8, 2004 Introduction to Algorithms L1.20
Kinds of analysesWorst-case: (usually)
• T(n) = maximum time of algorithm on any input of size n.
Average-case: (sometimes)• T(n) = expected time of algorithm
over all inputs of size n.• Need assumption of statistical
distribution of inputs.Best-case: (bogus)
• Cheat with a slow algorithm that works fast on some input. © 2001–4 by Charles E. Leiserson
September 8, 2004 Introduction to Algorithms L1.21
Machine-independent time
What is insertion sort’s worst-case time?• It depends on the speed of our computer:
• relative speed (on the same machine),• absolute speed (on different machines).
BIG IDEA:• Ignore machine-dependent constants.• Look at growth of T(n) as n →∞ .
“Asymptotic Analysis”“Asymptotic Analysis”© 2001–4 by Charles E. Leiserson
September 8, 2004 Introduction to Algorithms L1.22
Θ-notation
• Drop low-order terms; ignore leading constants.• Example: 3n3 + 90n2 – 5n + 6046 = Θ(n3)
Math:Θ(g(n)) = f (n) : there exist positive constants c1, c2, and
n0 such that 0 ≤ c1 g(n) ≤ f (n) ≤ c2 g(n)for all n ≥ n0
Engineering:
© 2001–4 by Charles E. Leiserson
September 8, 2004 Introduction to Algorithms L1.23
Asymptotic performance
n
T(n)
n0
• We shouldn’t ignore asymptotically slower algorithms, however.
• Real-world design situations often call for a careful balancing of engineering objectives.
• Asymptotic analysis is a useful tool to help to structure our thinking.
When n gets large enough, a Θ(n2) algorithm always beats a Θ(n3) algorithm.
© 2001–4 by Charles E. Leiserson
September 8, 2004 Introduction to Algorithms L1.24
Insertion sort analysisWorst case: Input reverse sorted.
( )∑=
Θ=Θ=n
jnjnT
2
2)()(
Average case: All permutations equally likely.
( )∑=
Θ=Θ=n
jnjnT
2
2)2/()(
Is insertion sort a fast sorting algorithm?• Moderately so, for small n.• Not at all, for large n.
[arithmetic series]
© 2001–4 by Charles E. Leiserson
September 8, 2004 Introduction to Algorithms L1.25
Merge sort
MERGE-SORT A[1 . . n]1. If n = 1, done.2. Recursively sort A[ 1 . . n/2 ]
and A[ n/2+1 . . n ] .3. “Merge” the 2 sorted lists.
Key subroutine: MERGE
© 2001–4 by Charles E. Leiserson
September 8, 2004 Introduction to Algorithms L1.26
Merging two sorted arrays
20
13
7
2
12
11
9
1
© 2001–4 by Charles E. Leiserson
September 8, 2004 Introduction to Algorithms L1.27
Merging two sorted arrays
20
13
7
2
12
11
9
1
1
© 2001–4 by Charles E. Leiserson
September 8, 2004 Introduction to Algorithms L1.28
Merging two sorted arrays
20
13
7
2
12
11
9
1
1
20
13
7
2
12
11
9
© 2001–4 by Charles E. Leiserson
September 8, 2004 Introduction to Algorithms L1.29
Merging two sorted arrays
20
13
7
2
12
11
9
1
1
20
13
7
2
12
11
9
2
© 2001–4 by Charles E. Leiserson
September 8, 2004 Introduction to Algorithms L1.30
Merging two sorted arrays
20
13
7
2
12
11
9
1
1
20
13
7
2
12
11
9
2
20
13
7
12
11
9
© 2001–4 by Charles E. Leiserson
September 8, 2004 Introduction to Algorithms L1.31
Merging two sorted arrays
20
13
7
2
12
11
9
1
1
20
13
7
2
12
11
9
2
20
13
7
12
11
9
7
© 2001–4 by Charles E. Leiserson
September 8, 2004 Introduction to Algorithms L1.32
Merging two sorted arrays
20
13
7
2
12
11
9
1
1
20
13
7
2
12
11
9
2
20
13
7
12
11
9
7
20
13
12
11
9
© 2001–4 by Charles E. Leiserson
September 8, 2004 Introduction to Algorithms L1.33
Merging two sorted arrays
20
13
7
2
12
11
9
1
1
20
13
7
2
12
11
9
2
20
13
7
12
11
9
7
20
13
12
11
9
9
© 2001–4 by Charles E. Leiserson
September 8, 2004 Introduction to Algorithms L1.34
Merging two sorted arrays
20
13
7
2
12
11
9
1
1
20
13
7
2
12
11
9
2
20
13
7
12
11
9
7
20
13
12
11
9
9
20
13
12
11
© 2001–4 by Charles E. Leiserson
September 8, 2004 Introduction to Algorithms L1.35
Merging two sorted arrays
20
13
7
2
12
11
9
1
1
20
13
7
2
12
11
9
2
20
13
7
12
11
9
7
20
13
12
11
9
9
20
13
12
11
11
© 2001–4 by Charles E. Leiserson
September 8, 2004 Introduction to Algorithms L1.36
Merging two sorted arrays
20
13
7
2
12
11
9
1
1
20
13
7
2
12
11
9
2
20
13
7
12
11
9
7
20
13
12
11
9
9
20
13
12
11
11
20
13
12
© 2001–4 by Charles E. Leiserson
September 8, 2004 Introduction to Algorithms L1.37
Merging two sorted arrays
20
13
7
2
12
11
9
1
1
20
13
7
2
12
11
9
2
20
13
7
12
11
9
7
20
13
12
11
9
9
20
13
12
11
11
20
13
12
12
© 2001–4 by Charles E. Leiserson
September 8, 2004 Introduction to Algorithms L1.38
Merging two sorted arrays
20
13
7
2
12
11
9
1
1
20
13
7
2
12
11
9
2
20
13
7
12
11
9
7
20
13
12
11
9
9
20
13
12
11
11
20
13
12
12
Time = Θ(n) to merge a total of n elements (linear time).
© 2001–4 by Charles E. Leiserson
September 8, 2004 Introduction to Algorithms L1.39
Analyzing merge sort
MERGE-SORT A[1 . . n]1. If n = 1, done.2. Recursively sort A[ 1 . . n/2 ]
and A[ n/2+1 . . n ] .3. “Merge” the 2 sorted lists
T(n)Θ(1)2T(n/2)
Θ(n)Abuse
Sloppiness: Should be T( n/2 ) + T( n/2 ) , but it turns out not to matter asymptotically.
© 2001–4 by Charles E. Leiserson
September 8, 2004 Introduction to Algorithms L1.40
Recurrence for merge sort
T(n) =Θ(1) if n = 1;2T(n/2) + Θ(n) if n > 1.
• We shall usually omit stating the base case when T(n) = Θ(1) for sufficiently small n, but only when it has no effect on the asymptotic solution to the recurrence.
• CLRS and Lecture 2 provide several ways to find a good upper bound on T(n).
© 2001–4 by Charles E. Leiserson
September 8, 2004 Introduction to Algorithms L1.41
Recursion treeSolve T(n) = 2T(n/2) + cn, where c > 0 is constant.
© 2001–4 by Charles E. Leiserson
September 8, 2004 Introduction to Algorithms L1.42
Recursion treeSolve T(n) = 2T(n/2) + cn, where c > 0 is constant.
T(n)
© 2001–4 by Charles E. Leiserson
September 8, 2004 Introduction to Algorithms L1.43
Recursion treeSolve T(n) = 2T(n/2) + cn, where c > 0 is constant.
T(n/2) T(n/2)
cn
© 2001–4 by Charles E. Leiserson
September 8, 2004 Introduction to Algorithms L1.44
Recursion treeSolve T(n) = 2T(n/2) + cn, where c > 0 is constant.
cn
T(n/4) T(n/4) T(n/4) T(n/4)
cn/2 cn/2
© 2001–4 by Charles E. Leiserson
September 8, 2004 Introduction to Algorithms L1.45
Recursion treeSolve T(n) = 2T(n/2) + cn, where c > 0 is constant.
cn
cn/4 cn/4 cn/4 cn/4
cn/2 cn/2
Θ(1)
…
© 2001–4 by Charles E. Leiserson
September 8, 2004 Introduction to Algorithms L1.46
Recursion treeSolve T(n) = 2T(n/2) + cn, where c > 0 is constant.
cn
cn/4 cn/4 cn/4 cn/4
cn/2 cn/2
Θ(1)
…
h = lg n
© 2001–4 by Charles E. Leiserson
September 8, 2004 Introduction to Algorithms L1.47
Recursion treeSolve T(n) = 2T(n/2) + cn, where c > 0 is constant.
cn
cn/4 cn/4 cn/4 cn/4
cn/2 cn/2
Θ(1)
…
h = lg n
cn
© 2001–4 by Charles E. Leiserson
September 8, 2004 Introduction to Algorithms L1.48
Recursion treeSolve T(n) = 2T(n/2) + cn, where c > 0 is constant.
cn
cn/4 cn/4 cn/4 cn/4
cn/2 cn/2
Θ(1)
…
h = lg n
cn
cn
© 2001–4 by Charles E. Leiserson
September 8, 2004 Introduction to Algorithms L1.49
Recursion treeSolve T(n) = 2T(n/2) + cn, where c > 0 is constant.
cn
cn/4 cn/4 cn/4 cn/4
cn/2 cn/2
Θ(1)
…
h = lg n
cn
cn
cn
…
© 2001–4 by Charles E. Leiserson
September 8, 2004 Introduction to Algorithms L1.50
Recursion treeSolve T(n) = 2T(n/2) + cn, where c > 0 is constant.
cn
cn/4 cn/4 cn/4 cn/4
cn/2 cn/2
Θ(1)
…
h = lg n
cn
cn
cn
#leaves = n Θ(n)
…
© 2001–4 by Charles E. Leiserson
September 8, 2004 Introduction to Algorithms L1.51
Recursion treeSolve T(n) = 2T(n/2) + cn, where c > 0 is constant.
cn
cn/4 cn/4 cn/4 cn/4
cn/2 cn/2
Θ(1)
…
h = lg n
cn
cn
cn
#leaves = n Θ(n)Total = Θ(n lg n)
…
© 2001–4 by Charles E. Leiserson
September 8, 2004 Introduction to Algorithms L1.52
Conclusions
• Θ(n lg n) grows more slowly than Θ(n2).• Therefore, merge sort asymptotically
beats insertion sort in the worst case.• In practice, merge sort beats insertion
sort for n > 30 or so.• Go test it out for yourself!
© 2001–4 by Charles E. Leiserson
Introduction to Algorithms6.046J/18.401J
Prof. Charles E. Leiserson
LECTURE 2Asymptotic Notation• O-, Ω-, and Θ-notationRecurrences• Substitution method• Iterating the recurrence• Recursion tree• Master method
September 13, 2004 Introduction to Algorithms L2.2
Asymptotic notation
We write f(n) = O(g(n)) if there exist constants c > 0, n0 > 0 such that 0 ≤ f(n) ≤ cg(n) for all n ≥ n0.
We write f(n) = O(g(n)) if there exist constants c > 0, n0 > 0 such that 0 ≤ f(n) ≤ cg(n) for all n ≥ n0.
O-notation (upper bounds):
© 2001–4 by Charles E. Leiserson
September 13, 2004 Introduction to Algorithms L2.3
Asymptotic notation
We write f(n) = O(g(n)) if there exist constants c > 0, n0 > 0 such that 0 ≤ f(n) ≤ cg(n) for all n ≥ n0.
We write f(n) = O(g(n)) if there exist constants c > 0, n0 > 0 such that 0 ≤ f(n) ≤ cg(n) for all n ≥ n0.
O-notation (upper bounds):
EXAMPLE: 2n2 = O(n3) (c = 1, n0 = 2)
© 2001–4 by Charles E. Leiserson
September 13, 2004 Introduction to Algorithms L2.4
Asymptotic notation
We write f(n) = O(g(n)) if there exist constants c > 0, n0 > 0 such that 0 ≤ f(n) ≤ cg(n) for all n ≥ n0.
We write f(n) = O(g(n)) if there exist constants c > 0, n0 > 0 such that 0 ≤ f(n) ≤ cg(n) for all n ≥ n0.
O-notation (upper bounds):
EXAMPLE: 2n2 = O(n3)
functions, not values
(c = 1, n0 = 2)
© 2001–4 by Charles E. Leiserson
September 13, 2004 Introduction to Algorithms L2.5
Asymptotic notation
We write f(n) = O(g(n)) if there exist constants c > 0, n0 > 0 such that 0 ≤ f(n) ≤ cg(n) for all n ≥ n0.
We write f(n) = O(g(n)) if there exist constants c > 0, n0 > 0 such that 0 ≤ f(n) ≤ cg(n) for all n ≥ n0.
O-notation (upper bounds):
EXAMPLE: 2n2 = O(n3)
functions, not values
funny, “one-way” equality
(c = 1, n0 = 2)
© 2001–4 by Charles E. Leiserson
September 13, 2004 Introduction to Algorithms L2.6
Set definition of O-notation
O(g(n)) = f(n) : there exist constants c > 0, n0 > 0 such that 0 ≤ f(n) ≤ cg(n)for all n ≥ n0
O(g(n)) = f(n) : there exist constants c > 0, n0 > 0 such that 0 ≤ f(n) ≤ cg(n)for all n ≥ n0
© 2001–4 by Charles E. Leiserson
September 13, 2004 Introduction to Algorithms L2.7
Set definition of O-notation
O(g(n)) = f(n) : there exist constants c > 0, n0 > 0 such that 0 ≤ f(n) ≤ cg(n)for all n ≥ n0
O(g(n)) = f(n) : there exist constants c > 0, n0 > 0 such that 0 ≤ f(n) ≤ cg(n)for all n ≥ n0
EXAMPLE: 2n2 ∈ O(n3)
© 2001–4 by Charles E. Leiserson
September 13, 2004 Introduction to Algorithms L2.8
Set definition of O-notation
O(g(n)) = f(n) : there exist constants c > 0, n0 > 0 such that 0 ≤ f(n) ≤ cg(n)for all n ≥ n0
O(g(n)) = f(n) : there exist constants c > 0, n0 > 0 such that 0 ≤ f(n) ≤ cg(n)for all n ≥ n0
EXAMPLE: 2n2 ∈ O(n3)(Logicians: λn.2n2 ∈ O(λn.n3), but it’s convenient to be sloppy, as long as we understand what’s really going on.)
© 2001–4 by Charles E. Leiserson
September 13, 2004 Introduction to Algorithms L2.9
Macro substitution
Convention: A set in a formula represents an anonymous function in the set.
© 2001–4 by Charles E. Leiserson
September 13, 2004 Introduction to Algorithms L2.10
Macro substitution
Convention: A set in a formula represents an anonymous function in the set.
f(n) = n3 + O(n2) means f(n) = n3 + h(n)for some h(n) ∈ O(n2) .
EXAMPLE:
© 2001–4 by Charles E. Leiserson
September 13, 2004 Introduction to Algorithms L2.11
Ω−notation (lower bounds)
O-notation is an upper-bound notation. It makes no sense to say f(n) is at least O(n2).
© 2001–4 by Charles E. Leiserson
September 13, 2004 Introduction to Algorithms L2.12
Ω−notation (lower bounds)
Ω(g(n)) = f(n) : there exist constants c > 0, n0 > 0 such that 0 ≤ cg(n) ≤ f(n)for all n ≥ n0
Ω(g(n)) = f(n) : there exist constants c > 0, n0 > 0 such that 0 ≤ cg(n) ≤ f(n)for all n ≥ n0
O-notation is an upper-bound notation. It makes no sense to say f(n) is at least O(n2).
© 2001–4 by Charles E. Leiserson
September 13, 2004 Introduction to Algorithms L2.13
Ω−notation (lower bounds)
Ω(g(n)) = f(n) : there exist constants c > 0, n0 > 0 such that 0 ≤ cg(n) ≤ f(n)for all n ≥ n0
Ω(g(n)) = f(n) : there exist constants c > 0, n0 > 0 such that 0 ≤ cg(n) ≤ f(n)for all n ≥ n0
EXAMPLE: )(lgnn Ω=© 2001–4 by Charles E. Leiserson
September 13, 2004 Introduction to Algorithms L2.14
Θ-notation (tight bounds)
Θ(g(n)) = O (g(n)) ∩ Ω(g(n))Θ(g(n)) = O (g(n)) ∩ Ω(g(n))
© 2001–4 by Charles E. Leiserson
September 13, 2004 Introduction to Algorithms L2.15
Θ-notation (tight bounds)
Θ(g(n)) = O (g(n)) ∩ Ω(g(n))Θ(g(n)) = O (g(n)) ∩ Ω(g(n))
EXAMPLE: )(221 22 nnn Θ=−
© 2001–4 by Charles E. Leiserson
September 13, 2004 Introduction to Algorithms L2.16
Θ-notation (tight bounds)
Θ(g(n)) = O (g(n)) ∩ Ω(g(n))Θ(g(n)) = O (g(n)) ∩ Ω(g(n))
EXAMPLE: )(2 2221 nnn Θ=−
Theorem. The leading constant and low-order terms don’t matter.
© 2001–4 by Charles E. Leiserson
September 13, 2004 Introduction to Algorithms L2.17
Solving recurrences
• The analysis of merge sort from Lecture 1required us to solve a recurrence.
• Recurrences are like solving integrals, differential equations, etc.oLearn a few tricks.
• Lecture 3: Applications of recurrences to divide-and-conquer algorithms.
© 2001–4 by Charles E. Leiserson
September 13, 2004 Introduction to Algorithms L2.18
Substitution method
1. Guess the form of the solution.2. Verify by induction.3. Solve for constants.
The most general method:
© 2001–4 by Charles E. Leiserson
September 13, 2004 Introduction to Algorithms L2.19
Substitution method
1. Guess the form of the solution.2. Verify by induction.3. Solve for constants.
The most general method:
EXAMPLE: T(n) = 4T(n/2) + n• [Assume that T(1) = Θ(1).]• Guess O(n3) . (Prove O and Ω separately.)• Assume that T(k) ≤ ck3 for k < n .• Prove T(n) ≤ cn3 by induction.
© 2001–4 by Charles E. Leiserson
September 13, 2004 Introduction to Algorithms L2.20
Example of substitution
3
33
3
3
))2/(()2/(
)2/(4)2/(4)(
cnnnccn
nncnnc
nnTnT
≤−−=
+=+≤
+=
desired – residual
whenever (c/2)n3 – n ≥ 0, for example, if c ≥ 2 and n ≥ 1.
desired
residual
© 2001–4 by Charles E. Leiserson
September 13, 2004 Introduction to Algorithms L2.21
Example (continued)• We must also handle the initial conditions,
that is, ground the induction with base cases.
• Base: T(n) = Θ(1) for all n < n0, where n0is a suitable constant.
• For 1 ≤ n < n0, we have “Θ(1)” ≤ cn3, if we pick c big enough.
© 2001–4 by Charles E. Leiserson
September 13, 2004 Introduction to Algorithms L2.22
Example (continued)• We must also handle the initial conditions,
that is, ground the induction with base cases.
• Base: T(n) = Θ(1) for all n < n0, where n0is a suitable constant.
• For 1 ≤ n < n0, we have “Θ(1)” ≤ cn3, if we pick c big enough.
This bound is not tight!© 2001–4 by Charles E. Leiserson
September 13, 2004 Introduction to Algorithms L2.23
A tighter upper bound?
We shall prove that T(n) = O(n2).
© 2001–4 by Charles E. Leiserson
September 13, 2004 Introduction to Algorithms L2.24
A tighter upper bound?
We shall prove that T(n) = O(n2).
Assume that T(k) ≤ ck2 for k < n:
)(
)2/(4)2/(4)(
2
2
2
nOncn
nncnnTnT
=+=
+≤+=
© 2001–4 by Charles E. Leiserson
September 13, 2004 Introduction to Algorithms L2.25
A tighter upper bound?
We shall prove that T(n) = O(n2).
Assume that T(k) ≤ ck2 for k < n:
)(
)2/(4)2/(4)(
2
2
2
nOncn
nncnnTnT
=+=
+≤+=
Wrong! We must prove the I.H.
© 2001–4 by Charles E. Leiserson
September 13, 2004 Introduction to Algorithms L2.26
A tighter upper bound?
We shall prove that T(n) = O(n2).
Assume that T(k) ≤ ck2 for k < n:
)(
)2/(4)2/(4)(
2
2
2
nOncn
nncnnTnT
=+=
+≤+=
Wrong! We must prove the I.H.
2
2 )(cn
ncn≤
−−=for no choice of c > 0. Lose!
[ desired – residual ]
© 2001–4 by Charles E. Leiserson
September 13, 2004 Introduction to Algorithms L2.27
A tighter upper bound!IDEA: Strengthen the inductive hypothesis.• Subtract a low-order term.Inductive hypothesis: T(k) ≤ c1k2 – c2k for k < n.
© 2001–4 by Charles E. Leiserson
September 13, 2004 Introduction to Algorithms L2.28
A tighter upper bound!IDEA: Strengthen the inductive hypothesis.• Subtract a low-order term.Inductive hypothesis: T(k) ≤ c1k2 – c2k for k < n.
T(n) = 4T(n/2) + n= 4(c1(n/2)2 – c2(n/2) + n= c1n2 – 2c2n + n= c1n2 – c2n – (c2n – n)≤ c1n2 – c2n if c2 ≥ 1.
© 2001–4 by Charles E. Leiserson
September 13, 2004 Introduction to Algorithms L2.29
A tighter upper bound!IDEA: Strengthen the inductive hypothesis.• Subtract a low-order term.Inductive hypothesis: T(k) ≤ c1k2 – c2k for k < n.
Pick c1 big enough to handle the initial conditions.
T(n) = 4T(n/2) + n= 4(c1(n/2)2 – c2(n/2) + n= c1n2 – 2c2n + n= c1n2 – c2n – (c2n – n)≤ c1n2 – c2n if c2 ≥ 1.
© 2001–4 by Charles E. Leiserson
September 13, 2004 Introduction to Algorithms L2.30
Recursion-tree method
• A recursion tree models the costs (time) of a recursive execution of an algorithm.
• The recursion-tree method can be unreliable, just like any method that uses ellipses (…).
• The recursion-tree method promotes intuition, however.
• The recursion tree method is good for generating guesses for the substitution method.
© 2001–4 by Charles E. Leiserson
September 13, 2004 Introduction to Algorithms L2.31
Example of recursion treeSolve T(n) = T(n/4) + T(n/2) + n2:
© 2001–4 by Charles E. Leiserson
September 13, 2004 Introduction to Algorithms L2.32
Example of recursion tree
T(n)
Solve T(n) = T(n/4) + T(n/2) + n2:
© 2001–4 by Charles E. Leiserson
September 13, 2004 Introduction to Algorithms L2.33
Example of recursion tree
T(n/4) T(n/2)
n2
Solve T(n) = T(n/4) + T(n/2) + n2:
© 2001–4 by Charles E. Leiserson
September 13, 2004 Introduction to Algorithms L2.34
Example of recursion treeSolve T(n) = T(n/4) + T(n/2) + n2:
n2
(n/4)2 (n/2)2
T(n/16) T(n/8) T(n/8) T(n/4)
© 2001–4 by Charles E. Leiserson
September 13, 2004 Introduction to Algorithms L2.35
Example of recursion tree
(n/16)2 (n/8)2 (n/8)2 (n/4)2
(n/4)2 (n/2)2
Θ(1)
…
Solve T(n) = T(n/4) + T(n/2) + n2:n2
© 2001–4 by Charles E. Leiserson
September 13, 2004 Introduction to Algorithms L2.36
Example of recursion treeSolve T(n) = T(n/4) + T(n/2) + n2:
(n/16)2 (n/8)2 (n/8)2 (n/4)2
(n/4)2 (n/2)2
Θ(1)
…
2nn2
© 2001–4 by Charles E. Leiserson
September 13, 2004 Introduction to Algorithms L2.37
Example of recursion treeSolve T(n) = T(n/4) + T(n/2) + n2:
(n/16)2 (n/8)2 (n/8)2 (n/4)2
(n/4)2 (n/2)2
Θ(1)
…
2165 n
2nn2
© 2001–4 by Charles E. Leiserson
September 13, 2004 Introduction to Algorithms L2.38
Example of recursion treeSolve T(n) = T(n/4) + T(n/2) + n2:
(n/16)2 (n/8)2 (n/8)2 (n/4)2
(n/4)2
Θ(1)
…
2165 n
2n
225625 n
n2
(n/2)2
…
© 2001–4 by Charles E. Leiserson
September 13, 2004 Introduction to Algorithms L2.39
Example of recursion treeSolve T(n) = T(n/4) + T(n/2) + n2:
(n/16)2 (n/8)2 (n/8)2 (n/4)2
(n/4)2
Θ(1)
…
2165 n
2n
225625 n
( ) ( )( ) 1 31652
165
1652 L++++n
…
Total == Θ(n2)
n2
(n/2)2
geometric series© 2001–4 by Charles E. Leiserson
September 13, 2004 Introduction to Algorithms L2.40
The master method
The master method applies to recurrences of the form
T(n) = a T(n/b) + f (n) , where a ≥ 1, b > 1, and f is asymptotically positive.
© 2001–4 by Charles E. Leiserson
September 13, 2004 Introduction to Algorithms L2.41
Three common casesCompare f (n) with nlogba:1. f (n) = O(nlogba – ε) for some constant ε > 0.
• f (n) grows polynomially slower than nlogba
(by an nε factor).Solution: T(n) = Θ(nlogba) .
© 2001–4 by Charles E. Leiserson
September 13, 2004 Introduction to Algorithms L2.42
Three common casesCompare f (n) with nlogba:1. f (n) = O(nlogba – ε) for some constant ε > 0.
• f (n) grows polynomially slower than nlogba
(by an nε factor).Solution: T(n) = Θ(nlogba) .
2. f (n) = Θ(nlogba lgkn) for some constant k ≥ 0.• f (n) and nlogba grow at similar rates.Solution: T(n) = Θ(nlogba lgk+1n) .
© 2001–4 by Charles E. Leiserson
September 13, 2004 Introduction to Algorithms L2.43
Three common cases (cont.)Compare f (n) with nlogba:
3. f (n) = Ω(nlogba + ε) for some constant ε > 0.• f (n) grows polynomially faster than nlogba (by
an nε factor),and f (n) satisfies the regularity condition that a f (n/b) ≤ c f (n) for some constant c < 1.Solution: T(n) = Θ( f (n)) .
© 2001–4 by Charles E. Leiserson
September 13, 2004 Introduction to Algorithms L2.44
Examples
EX. T(n) = 4T(n/2) + na = 4, b = 2 ⇒ nlogba = n2; f (n) = n.CASE 1: f (n) = O(n2 – ε) for ε = 1.∴ T(n) = Θ(n2).
© 2001–4 by Charles E. Leiserson
September 13, 2004 Introduction to Algorithms L2.45
Examples
EX. T(n) = 4T(n/2) + na = 4, b = 2 ⇒ nlogba = n2; f (n) = n.CASE 1: f (n) = O(n2 – ε) for ε = 1.∴ T(n) = Θ(n2).
EX. T(n) = 4T(n/2) + n2
a = 4, b = 2 ⇒ nlogba = n2; f (n) = n2.CASE 2: f (n) = Θ(n2lg0n), that is, k = 0.∴ T(n) = Θ(n2lg n).
© 2001–4 by Charles E. Leiserson
September 13, 2004 Introduction to Algorithms L2.46
Examples
EX. T(n) = 4T(n/2) + n3
a = 4, b = 2 ⇒ nlogba = n2; f (n) = n3.CASE 3: f (n) = Ω(n2 + ε) for ε = 1and 4(n/2)3 ≤ cn3 (reg. cond.) for c = 1/2.∴ T(n) = Θ(n3).
© 2001–4 by Charles E. Leiserson
September 13, 2004 Introduction to Algorithms L2.47
Examples
EX. T(n) = 4T(n/2) + n3
a = 4, b = 2 ⇒ nlogba = n2; f (n) = n3.CASE 3: f (n) = Ω(n2 + ε) for ε = 1and 4(n/2)3 ≤ cn3 (reg. cond.) for c = 1/2.∴ T(n) = Θ(n3).
EX. T(n) = 4T(n/2) + n2/lgna = 4, b = 2 ⇒ nlogba = n2; f (n) = n2/lgn.Master method does not apply. In particular, for every constant ε > 0, we have nε = ω(lgn).
© 2001–4 by Charles E. Leiserson
September 13, 2004 Introduction to Algorithms L2.48
f (n/b)
Idea of master theorem
f (n/b) f (n/b)
Τ (1)
…Recursion tree:
…f (n) a
f (n/b2)f (n/b2) f (n/b2)…a
© 2001–4 by Charles E. Leiserson
September 13, 2004 Introduction to Algorithms L2.49
f (n/b)
Idea of master theorem
f (n/b) f (n/b)
Τ (1)
…Recursion tree:
…f (n) a
f (n/b2)f (n/b2) f (n/b2)…a
f (n)
a f (n/b)
a2 f (n/b2)
…
© 2001–4 by Charles E. Leiserson
September 13, 2004 Introduction to Algorithms L2.50
f (n/b)
Idea of master theorem
f (n/b) f (n/b)
Τ (1)
…Recursion tree:
…f (n) a
f (n/b2)f (n/b2) f (n/b2)…ah = logbn
f (n)
a f (n/b)
a2 f (n/b2)
…
© 2001–4 by Charles E. Leiserson
September 13, 2004 Introduction to Algorithms L2.51
nlogbaΤ (1)
f (n/b)
Idea of master theorem
f (n/b) f (n/b)
Τ (1)
…Recursion tree:
…f (n) a
f (n/b2)f (n/b2) f (n/b2)…ah = logbn
f (n)
a f (n/b)
a2 f (n/b2)
#leaves = ah
= alogbn
= nlogba
…
© 2001–4 by Charles E. Leiserson
September 13, 2004 Introduction to Algorithms L2.52
f (n/b)
Idea of master theorem
f (n/b) f (n/b)
Τ (1)
…Recursion tree:
…f (n) a
f (n/b2)f (n/b2) f (n/b2)…ah = logbn
f (n)
a f (n/b)
a2 f (n/b2)
CASE 1: The weight increases geometrically from the root to the leaves. The leaves hold a constant fraction of the total weight.
CASE 1: The weight increases geometrically from the root to the leaves. The leaves hold a constant fraction of the total weight.
Θ(nlogba)
…
nlogbaΤ (1)
© 2001–4 by Charles E. Leiserson
September 13, 2004 Introduction to Algorithms L2.53
f (n/b)
Idea of master theorem
f (n/b) f (n/b)
Τ (1)
…Recursion tree:
…f (n) a
f (n/b2)f (n/b2) f (n/b2)…ah = logbn
f (n)
a f (n/b)
a2 f (n/b2)
CASE 2: (k = 0) The weight is approximately the same on each of the logbn levels.
CASE 2: (k = 0) The weight is approximately the same on each of the logbn levels.
Θ(nlogbalg n)
…
nlogbaΤ (1)
© 2001–4 by Charles E. Leiserson
September 13, 2004 Introduction to Algorithms L2.54
f (n/b)
Idea of master theorem
f (n/b) f (n/b)
Τ (1)
…Recursion tree:
…f (n) a
f (n/b2)f (n/b2) f (n/b2)…ah = logbn
f (n)
a f (n/b)
a2 f (n/b2)
…CASE 3: The weight decreases geometrically from the root to the leaves. The root holds a constant fraction of the total weight.
CASE 3: The weight decreases geometrically from the root to the leaves. The root holds a constant fraction of the total weight.
nlogbaΤ (1)
Θ( f (n))© 2001–4 by Charles E. Leiserson
September 13, 2004 Introduction to Algorithms L2.55
Appendix: geometric series
1
11 2x
xx−
=+++ L for |x| < 1
1
111
2x
xxxxn
n−
−=+++++
L for x ≠ 1
Return to last slide viewed.
© 2001–4 by Charles E. Leiserson
Introduction to Algorithms6.046J/18.401J
Prof. Charles E. Leiserson
LECTURE 3Divide and conquer• Binary search• Powering a number• Fibonacci numbers• Matrix multiplication• Strassen’s algorithm• VLSI tree layout
September 15, 2004 Introduction to Algorithms L3.2
The divide-and-conquer design paradigm
1. Divide the problem (instance) into subproblems.
2. Conquer the subproblems by solving them recursively.
3. Combine subproblem solutions.
© 2001–4 by Charles E. Leiserson
September 15, 2004 Introduction to Algorithms L3.3
Merge sort
1. Divide: Trivial.2. Conquer: Recursively sort 2 subarrays.3. Combine: Linear-time merge.
© 2001–4 by Charles E. Leiserson
September 15, 2004 Introduction to Algorithms L3.4
Merge sort
1. Divide: Trivial.2. Conquer: Recursively sort 2 subarrays.3. Combine: Linear-time merge.
T(n) = 2 T(n/2) + Θ(n)
# subproblemssubproblem size
work dividing and combining
© 2001–4 by Charles E. Leiserson
September 15, 2004 Introduction to Algorithms L3.5
Master theorem (reprise)T(n) = a T(n/b) + f (n)
CASE 1: f (n) = O(nlogba – ε), constant ε > 0⇒ T(n) = Θ(nlogba) .
CASE 2: f (n) = Θ(nlogba lgkn), constant k ≥ 0⇒ T(n) = Θ(nlogba lgk+1n) .
CASE 3: f (n) = Ω(nlogba + ε ), constant ε > 0, and regularity condition
⇒ T(n) = Θ( f (n)) .
© 2001–4 by Charles E. Leiserson
September 15, 2004 Introduction to Algorithms L3.6
Master theorem (reprise)T(n) = a T(n/b) + f (n)
CASE 1: f (n) = O(nlogba – ε), constant ε > 0⇒ T(n) = Θ(nlogba) .
CASE 2: f (n) = Θ(nlogba lgkn), constant k ≥ 0⇒ T(n) = Θ(nlogba lgk+1n) .
CASE 3: f (n) = Ω(nlogba + ε ), constant ε > 0, and regularity condition
⇒ T(n) = Θ( f (n)) .Merge sort: a = 2, b = 2 ⇒ nlogba = nlog22 = n
⇒ CASE 2 (k = 0) ⇒ T(n) = Θ(n lg n) . © 2001–4 by Charles E. Leiserson
September 15, 2004 Introduction to Algorithms L3.7
Binary search
Find an element in a sorted array:1. Divide: Check middle element.2. Conquer: Recursively search 1 subarray.3. Combine: Trivial.
© 2001–4 by Charles E. Leiserson
September 15, 2004 Introduction to Algorithms L3.8
Binary search
Example: Find 9
3 5 7 8 9 12 15
Find an element in a sorted array:1. Divide: Check middle element.2. Conquer: Recursively search 1 subarray.3. Combine: Trivial.
© 2001–4 by Charles E. Leiserson
September 15, 2004 Introduction to Algorithms L3.9
Binary search
Example: Find 9
3 5 7 8 9 12 15
Find an element in a sorted array:1. Divide: Check middle element.2. Conquer: Recursively search 1 subarray.3. Combine: Trivial.
© 2001–4 by Charles E. Leiserson
September 15, 2004 Introduction to Algorithms L3.10
Binary search
Example: Find 9
3 5 7 8 9 12 15
Find an element in a sorted array:1. Divide: Check middle element.2. Conquer: Recursively search 1 subarray.3. Combine: Trivial.
© 2001–4 by Charles E. Leiserson
September 15, 2004 Introduction to Algorithms L3.11
Binary search
Example: Find 9
3 5 7 8 9 12 15
Find an element in a sorted array:1. Divide: Check middle element.2. Conquer: Recursively search 1 subarray.3. Combine: Trivial.
© 2001–4 by Charles E. Leiserson
September 15, 2004 Introduction to Algorithms L3.12
Binary search
Example: Find 9
3 5 7 8 9 12 15
Find an element in a sorted array:1. Divide: Check middle element.2. Conquer: Recursively search 1 subarray.3. Combine: Trivial.
© 2001–4 by Charles E. Leiserson
September 15, 2004 Introduction to Algorithms L3.13
Binary search
Find an element in a sorted array:1. Divide: Check middle element.2. Conquer: Recursively search 1 subarray.3. Combine: Trivial.
Example: Find 9
3 5 7 8 9 12 15
© 2001–4 by Charles E. Leiserson
September 15, 2004 Introduction to Algorithms L3.14
Recurrence for binary search
T(n) = 1 T(n/2) + Θ(1)
# subproblemssubproblem size
work dividing and combining
© 2001–4 by Charles E. Leiserson
September 15, 2004 Introduction to Algorithms L3.15
Recurrence for binary search
T(n) = 1 T(n/2) + Θ(1)
# subproblemssubproblem size
work dividing and combining
nlogba = nlog21 = n0 = 1 ⇒ CASE 2 (k = 0)⇒ T(n) = Θ(lg n) .
© 2001–4 by Charles E. Leiserson
September 15, 2004 Introduction to Algorithms L3.16
Powering a number
Problem: Compute a n, where n ∈ N.
Naive algorithm: Θ(n).
© 2001–4 by Charles E. Leiserson
September 15, 2004 Introduction to Algorithms L3.17
Powering a number
Problem: Compute a n, where n ∈ N.
a n =a n/2 ⋅ a n/2 if n is even;a (n–1)/2 ⋅ a (n–1)/2 ⋅ a if n is odd.
Divide-and-conquer algorithm:
Naive algorithm: Θ(n).
© 2001–4 by Charles E. Leiserson
September 15, 2004 Introduction to Algorithms L3.18
Powering a number
Problem: Compute a n, where n ∈ N.
a n =a n/2 ⋅ a n/2 if n is even;a (n–1)/2 ⋅ a (n–1)/2 ⋅ a if n is odd.
Divide-and-conquer algorithm:
T(n) = T(n/2) + Θ(1) ⇒ T(n) = Θ(lg n) .
Naive algorithm: Θ(n).
© 2001–4 by Charles E. Leiserson
September 15, 2004 Introduction to Algorithms L3.19
Fibonacci numbersRecursive definition:
Fn =0 if n = 0;
Fn–1 + Fn–2 if n ≥ 2.1 if n = 1;
0 1 1 2 3 5 8 13 21 34 L
© 2001–4 by Charles E. Leiserson
September 15, 2004 Introduction to Algorithms L3.20
Fibonacci numbersRecursive definition:
Fn =0 if n = 0;
Fn–1 + Fn–2 if n ≥ 2.1 if n = 1;
0 1 1 2 3 5 8 13 21 34 L
Naive recursive algorithm: Ω(φ n)(exponential time), where φ =is the golden ratio.
2/)51( +
© 2001–4 by Charles E. Leiserson
September 15, 2004 Introduction to Algorithms L3.21
Computing Fibonacci numbers
Bottom-up: • Compute F0, F1, F2, …, Fn in order, forming
each number by summing the two previous.• Running time: Θ(n).
© 2001–4 by Charles E. Leiserson
September 15, 2004 Introduction to Algorithms L3.22
Computing Fibonacci numbers
Naive recursive squaring:Fn = φ n/ rounded to the nearest integer.5
• Recursive squaring: Θ(lg n) time. • This method is unreliable, since floating-point
arithmetic is prone to round-off errors.
Bottom-up: • Compute F0, F1, F2, …, Fn in order, forming
each number by summing the two previous.• Running time: Θ(n).
© 2001–4 by Charles E. Leiserson
September 15, 2004 Introduction to Algorithms L3.23
Recursive squaringn
FFFF
nn
nn
=
−
+
0111
1
1Theorem: .
© 2001–4 by Charles E. Leiserson
September 15, 2004 Introduction to Algorithms L3.24
Recursive squaringn
FFFF
nn
nn
=
−
+
0111
1
1Theorem: .
Algorithm: Recursive squaring.Time = Θ(lg n) .
© 2001–4 by Charles E. Leiserson
September 15, 2004 Introduction to Algorithms L3.25
Recursive squaringn
FFFF
nn
nn
=
−
+
0111
1
1Theorem: .
Proof of theorem. (Induction on n.)
Base (n = 1): .1
0111
01
12
=
FFFF
Algorithm: Recursive squaring.Time = Θ(lg n) .
© 2001–4 by Charles E. Leiserson
September 15, 2004 Introduction to Algorithms L3.26
Recursive squaring
.
.
Inductive step (n ≥ 2):
n
nFFFF
FFFF
nn
nn
nn
nn
=
⋅−
=
⋅
=
−−
−
−
+
0111
01111
0111
0111
21
1
1
1
© 2001–4 by Charles E. Leiserson
September 15, 2004 Introduction to Algorithms L3.27
Matrix multiplication
⋅
=
nnnn
n
n
nnnn
n
n
nnnn
n
n
bbb
bbbbbb
aaa
aaaaaa
ccc
cccccc
L
MOMM
L
L
L
MOMM
L
L
L
MOMM
L
L
21
22221
11211
21
22221
11211
21
22221
11211
∑=
⋅=n
kkjikij bac
1
Input: A = [aij], B = [bij].Output: C = [cij] = A⋅ B. i, j = 1, 2,… , n.
© 2001–4 by Charles E. Leiserson
September 15, 2004 Introduction to Algorithms L3.28
Standard algorithm
for i ← 1 to ndo for j ← 1 to n
do cij ← 0for k ← 1 to n
do cij ← cij + aik⋅ bkj
© 2001–4 by Charles E. Leiserson
September 15, 2004 Introduction to Algorithms L3.29
Standard algorithm
for i ← 1 to ndo for j ← 1 to n
do cij ← 0for k ← 1 to n
do cij ← cij + aik⋅ bkj
Running time = Θ(n3)
© 2001–4 by Charles E. Leiserson
September 15, 2004 Introduction to Algorithms L3.30
Divide-and-conquer algorithm
n×n matrix = 2×2 matrix of (n/2)×(n/2) submatrices:IDEA:
⋅
=
hgfe
dcba
utsr
C = A ⋅ Br = ae + bgs = af + bht = ce + dgu = cf + dh
8 mults of (n/2)×(n/2) submatrices4 adds of (n/2)×(n/2) submatrices
© 2001–4 by Charles E. Leiserson
September 15, 2004 Introduction to Algorithms L3.31
Divide-and-conquer algorithm
n×n matrix = 2×2 matrix of (n/2)×(n/2) submatrices:IDEA:
⋅
=
hgfe
dcba
utsr
C = A ⋅ Br = ae + bgs = af + bht = ce + dhu = cf + dg
8 mults of (n/2)×(n/2) submatrices4 adds of (n/2)×(n/2) submatrices^
recursive
© 2001–4 by Charles E. Leiserson
September 15, 2004 Introduction to Algorithms L3.32
Analysis of D&C algorithm
# submatricessubmatrix size
work adding submatrices
T(n) = 8 T(n/2) + Θ(n2)
© 2001–4 by Charles E. Leiserson
September 15, 2004 Introduction to Algorithms L3.33
Analysis of D&C algorithm
nlogba = nlog28 = n3 ⇒ CASE 1 ⇒ T(n) = Θ(n3).
# submatricessubmatrix size
work adding submatrices
T(n) = 8 T(n/2) + Θ(n2)
© 2001–4 by Charles E. Leiserson
September 15, 2004 Introduction to Algorithms L3.34
Analysis of D&C algorithm
nlogba = nlog28 = n3 ⇒ CASE 1 ⇒ T(n) = Θ(n3).
No better than the ordinary algorithm.
# submatricessubmatrix size
work adding submatrices
T(n) = 8 T(n/2) + Θ(n2)
© 2001–4 by Charles E. Leiserson
September 15, 2004 Introduction to Algorithms L3.35
Strassen’s idea• Multiply 2×2 matrices with only 7 recursive mults.
© 2001–4 by Charles E. Leiserson
September 15, 2004 Introduction to Algorithms L3.36
Strassen’s idea• Multiply 2×2 matrices with only 7 recursive mults.
P1 = a ⋅ ( f – h)P2 = (a + b) ⋅ hP3 = (c + d) ⋅ eP4 = d ⋅ (g – e)P5 = (a + d) ⋅ (e + h)P6 = (b – d) ⋅ (g + h)P7 = (a – c) ⋅ (e + f )
© 2001–4 by Charles E. Leiserson
September 15, 2004 Introduction to Algorithms L3.37
Strassen’s idea• Multiply 2×2 matrices with only 7 recursive mults.
P1 = a ⋅ ( f – h)P2 = (a + b) ⋅ hP3 = (c + d) ⋅ eP4 = d ⋅ (g – e)P5 = (a + d) ⋅ (e + h)P6 = (b – d) ⋅ (g + h)P7 = (a – c) ⋅ (e + f )
r = P5 + P4 – P2 + P6s = P1 + P2t = P3 + P4u = P5 + P1 – P3 – P7
© 2001–4 by Charles E. Leiserson
September 15, 2004 Introduction to Algorithms L3.38
7 mults, 18 adds/subs.Note: No reliance on commutativity of mult!
7 mults, 18 adds/subs.Note: No reliance on commutativity of mult!
Strassen’s idea• Multiply 2×2 matrices with only 7 recursive mults.
P1 = a ⋅ ( f – h)P2 = (a + b) ⋅ hP3 = (c + d) ⋅ eP4 = d ⋅ (g – e)P5 = (a + d) ⋅ (e + h)P6 = (b – d) ⋅ (g + h)P7 = (a – c) ⋅ (e + f )
r = P5 + P4 – P2 + P6s = P1 + P2t = P3 + P4u = P5 + P1 – P3 – P7
© 2001–4 by Charles E. Leiserson
September 15, 2004 Introduction to Algorithms L3.39
Strassen’s idea• Multiply 2×2 matrices with only 7 recursive mults.
P1 = a ⋅ ( f – h)P2 = (a + b) ⋅ hP3 = (c + d) ⋅ eP4 = d ⋅ (g – e)P5 = (a + d) ⋅ (e + h)P6 = (b – d) ⋅ (g + h)P7 = (a – c) ⋅ (e + f )
r = P5 + P4 – P2 + P6= (a + d) (e + h)
+ d (g – e) – (a + b) h+ (b – d) (g + h)
= ae + ah + de + dh + dg –de – ah – bh+ bg + bh – dg – dh
= ae + bg© 2001–4 by Charles E. Leiserson
September 15, 2004 Introduction to Algorithms L3.40
Strassen’s algorithm1. Divide: Partition A and B into
(n/2)×(n/2) submatrices. Form terms to be multiplied using + and – .
2. Conquer: Perform 7 multiplications of (n/2)×(n/2) submatrices recursively.
3. Combine: Form C using + and – on (n/2)×(n/2) submatrices.
© 2001–4 by Charles E. Leiserson
September 15, 2004 Introduction to Algorithms L3.41
Strassen’s algorithm1. Divide: Partition A and B into
(n/2)×(n/2) submatrices. Form terms to be multiplied using + and – .
2. Conquer: Perform 7 multiplications of (n/2)×(n/2) submatrices recursively.
3. Combine: Form C using + and – on (n/2)×(n/2) submatrices.
T(n) = 7 T(n/2) + Θ(n2)
© 2001–4 by Charles E. Leiserson
September 15, 2004 Introduction to Algorithms L3.42
Analysis of StrassenT(n) = 7 T(n/2) + Θ(n2)
© 2001–4 by Charles E. Leiserson
September 15, 2004 Introduction to Algorithms L3.43
Analysis of StrassenT(n) = 7 T(n/2) + Θ(n2)
nlogba = nlog27 ≈ n2.81 ⇒ CASE 1 ⇒ T(n) = Θ(nlg 7).
© 2001–4 by Charles E. Leiserson
September 15, 2004 Introduction to Algorithms L3.44
Analysis of StrassenT(n) = 7 T(n/2) + Θ(n2)
nlogba = nlog27 ≈ n2.81 ⇒ CASE 1 ⇒ T(n) = Θ(nlg 7).
The number 2.81 may not seem much smaller than 3, but because the difference is in the exponent, the impact on running time is significant. In fact, Strassen’s algorithm beats the ordinary algorithm on today’s machines for n ≥ 32 or so.
© 2001–4 by Charles E. Leiserson
September 15, 2004 Introduction to Algorithms L3.45
Analysis of StrassenT(n) = 7 T(n/2) + Θ(n2)
nlogba = nlog27 ≈ n2.81 ⇒ CASE 1 ⇒ T(n) = Θ(nlg 7).
Best to date (of theoretical interest only): Θ(n2.376L).
The number 2.81 may not seem much smaller than 3, but because the difference is in the exponent, the impact on running time is significant. In fact, Strassen’s algorithm beats the ordinary algorithm on today’s machines for n ≥ 32 or so.
© 2001–4 by Charles E. Leiserson
September 15, 2004 Introduction to Algorithms L3.46
VLSI layoutProblem: Embed a complete binary tree with n leaves in a grid using minimal area.
© 2001–4 by Charles E. Leiserson
September 15, 2004 Introduction to Algorithms L3.47
VLSI layoutProblem: Embed a complete binary tree with n leaves in a grid using minimal area.
H(n)
W(n)
© 2001–4 by Charles E. Leiserson
September 15, 2004 Introduction to Algorithms L3.48
VLSI layoutProblem: Embed a complete binary tree with n leaves in a grid using minimal area.
H(n)
W(n)
H(n) = H(n/2) + Θ(1)= Θ(lg n)
© 2001–4 by Charles E. Leiserson
September 15, 2004 Introduction to Algorithms L3.49
VLSI layoutProblem: Embed a complete binary tree with n leaves in a grid using minimal area.
H(n)
W(n)
H(n) = H(n/2) + Θ(1)= Θ(lg n)
W(n) = 2W(n/2) + Θ(1)= Θ(n)
© 2001–4 by Charles E. Leiserson
September 15, 2004 Introduction to Algorithms L3.50
VLSI layoutProblem: Embed a complete binary tree with n leaves in a grid using minimal area.
H(n)
W(n)
H(n) = H(n/2) + Θ(1)= Θ(lg n)
W(n) = 2W(n/2) + Θ(1)= Θ(n)
Area = Θ(n lg n)© 2001–4 by Charles E. Leiserson
September 15, 2004 Introduction to Algorithms L3.51
H-tree embeddingL(n)
L(n)
© 2001–4 by Charles E. Leiserson
September 15, 2004 Introduction to Algorithms L3.52
H-tree embeddingL(n)
L(n)
L(n/4) L(n/4)Θ(1)© 2001–4 by Charles E. Leiserson
September 15, 2004 Introduction to Algorithms L3.53
H-tree embeddingL(n)
L(n)
L(n/4) L(n/4)Θ(1)
L(n) = 2L(n/4) + Θ(1)= Θ( )n
Area = Θ(n)
© 2001–4 by Charles E. Leiserson
September 15, 2004 Introduction to Algorithms L3.54
Conclusion
• Divide and conquer is just one of several powerful techniques for algorithm design.
• Divide-and-conquer algorithms can be analyzed using recurrences and the master method (so practice this math).
• The divide-and-conquer strategy often leads to efficient algorithms.
© 2001–4 by Charles E. Leiserson
Introduction to Algorithms6.046J/18.401J
Lecture 4Prof. Piotr Indyk
September 20, 2004 (c) Piotr Indyk & Charles Leiserson L4.2
Today
• Randomized algorithms: algorithms that flip coins– Matrix product checker: is AB=C ?– Quicksort:
• Example of divide and conquer• Fast and practical sorting algorithm• Other applications on Wednesday
September 20, 2004 (c) Piotr Indyk & Charles Leiserson L4.3
Randomized Algorithms
• Algorithms that make random decisions• That is:
– Can generate a random number x from some range 1…R
– Make decisions based on the value of x• Why would it make sense ?
September 20, 2004 (c) Piotr Indyk & Charles Leiserson L4.4
Two cups, one coin
• If you always choose a fixed cup, the adversary will put the coin in the other one, so the expected payoff = $0
• If you choose a random cup, the expected payoff = $0.5
September 20, 2004 (c) Piotr Indyk & Charles Leiserson L4.5
Randomized Algorithms
• Two basic types:– Typically fast (but sometimes slow):
Las Vegas– Typically correct (but sometimes output
garbage): Monte Carlo• The probabilities are defined by the random
numbers of the algorithm! (not by random choices of the problem instance)
September 20, 2004 (c) Piotr Indyk & Charles Leiserson L4.6
Matrix Product
• Compute C=A×B– Simple algorithm: O(n3) time– Multiply two 2×2 matrices using 7 mult. →O(n2.81…) time [Strassen’69]
– Multiply two 70 × 70 matrices using 143640multiplications → O(n2.795…) time [Pan’78]
– …– O(n2.376…) [Coppersmith-Winograd]
September 20, 2004 (c) Piotr Indyk & Charles Leiserson L4.7
Matrix Product Checker
• Given: n×n matrices A,B,C• Goal: is A×B=C ?• We will see an O(n2) algorithm that:
– If answer=YES, then Pr[output=YES]=1– If answer=NO, then Pr[output=YES] ≤ ½
September 20, 2004 (c) Piotr Indyk & Charles Leiserson L4.8
The algorithm
• Algorithm:– Choose a random binary vector x[1…n] ,
such that Pr[xi=1]=½ , i=1…n– Check if ABx=Cx
• Does it run in O(n2) time ?– YES, because ABx = A(Bx)
September 20, 2004 (c) Piotr Indyk & Charles Leiserson L4.9
Correctness
• Let D=AB, need to check if D=C• What if D=C ?
– Then Dx=Cx ,so the output is YES• What if D≠C ?
– Presumably there exists x such that Dx≠Cx
– We need to show there are many such x
September 20, 2004 (c) Piotr Indyk & Charles Leiserson L4.10
D≠C
≠?
September 20, 2004 (c) Piotr Indyk & Charles Leiserson L4.11
Vector product
• Consider vectors d≠c (say, di≠ci)• Choose a random binary x• We have dx=cx iff (d-c)x=0• Pr[(d-c)x=0]= ?
(d-c): d1-c1 d2-c2 di-ci dn-cn… …x: x1 x2 xi xn… … = Σj≠i(dj-cj)xj + (di-ci)xi
September 20, 2004 (c) Piotr Indyk & Charles Leiserson L4.12
Analysis, ctd.
• If xi=0, then (c-d)x=S1
• If xi=1, then (c-d)x=S2≠S1
• So, ≥1 of the choices gives (c-d)x≠0→ Pr[cx=dx] ≤ ½
September 20, 2004 (c) Piotr Indyk & Charles Leiserson L4.13
Matrix Product Checker
• Is A×B=C ?• We have an algorithm that:
– If answer=YES, then Pr[output=YES]=1– If answer=NO, then Pr[output=YES] ≤ ½
• What if we want to reduce ½ to ¼ ?– Run the algorithm twice, using independent random numbers– Output YES only if both runs say YES
• Analysis:– If answer=YES, then Pr[output1=YES, output2=YES ]=1– If answer=NO, then
Pr[output=YES] = Pr[output1=YES, output2=YES]= Pr[output1=YES]*Pr[output2=YES]≤ ¼
September 20, 2004 (c) Piotr Indyk & Charles Leiserson L4.14
Quicksort
• Proposed by C.A.R. Hoare in 1962.• Divide-and-conquer algorithm.• Sorts “in place” (like insertion sort, but not
like merge sort).• Very practical (with tuning).• Can be viewed as a randomized Las Vegas
algorithm
September 20, 2004 (c) Piotr Indyk & Charles Leiserson L4.15
Quicksort an n-element array:1. Divide: Partition the array into two subarrays
around a pivot x such that elements in lower subarray ≤ x ≤ elements in upper subarray.
2. Conquer: Recursively sort the two subarrays.3. Combine: Trivial.
Divide and conquer
xx
Key: Linear-time partitioning subroutine.
≤ x≤ x≤ x≤ x xx ≥ x≥ x
September 20, 2004 (c) Piotr Indyk & Charles Leiserson L4.16
Pseudocode for quicksortQUICKSORT(A, p, r)
if p < rthen q ← PARTITION(A, p, r)
QUICKSORT(A, p, q–1)QUICKSORT(A, q+1, r)
Initial call: QUICKSORT(A, 1, n)
September 20, 2004 (c) Piotr Indyk & Charles Leiserson L4.17
xx
Partitioning subroutinePARTITION(A, p, r) ⊳ A[p . . r]
x ← A[p] ⊳ pivot = A[p]i ← pfor j ← p + 1 to r
do if A[ j] ≤ xthen i ← i + 1
exchange A[i] ↔ A[ j]exchange A[p] ↔ A[i]return i
≤ x≤ x ≥ x≥ x ??p i rj
Invariant:
September 20, 2004 (c) Piotr Indyk & Charles Leiserson L4.18
Example of partitioning
i j66 1010 1313 55 88 33 22 1111
September 20, 2004 (c) Piotr Indyk & Charles Leiserson L4.19
Example of partitioning
i j66 1010 1313 55 88 33 22 1111
September 20, 2004 (c) Piotr Indyk & Charles Leiserson L4.20
Example of partitioning
i j66 1010 1313 55 88 33 22 1111
September 20, 2004 (c) Piotr Indyk & Charles Leiserson L4.21
Example of partitioning
66 1010 1313 55 88 33 22 1111
i j66 55 1313 1010 88 33 22 1111
September 20, 2004 (c) Piotr Indyk & Charles Leiserson L4.22
Example of partitioning
66 1010 1313 55 88 33 22 1111
i j66 55 1313 1010 88 33 22 1111
September 20, 2004 (c) Piotr Indyk & Charles Leiserson L4.23
Example of partitioning
66 1010 1313 55 88 33 22 1111
i j66 55 1313 1010 88 33 22 1111
September 20, 2004 (c) Piotr Indyk & Charles Leiserson L4.24
Example of partitioning
66 1010 1313 55 88 33 22 1111
i j66 55 33 1010 88 1313 22 1111
66 55 1313 1010 88 33 22 1111
September 20, 2004 (c) Piotr Indyk & Charles Leiserson L4.25
Example of partitioning
66 1010 1313 55 88 33 22 1111
i j66 55 33 1010 88 1313 22 1111
66 55 1313 1010 88 33 22 1111
September 20, 2004 (c) Piotr Indyk & Charles Leiserson L4.26
Example of partitioning
66 1010 1313 55 88 33 22 1111
66 55 33 1010 88 1313 22 1111
66 55 1313 1010 88 33 22 1111
i j66 55 33 22 88 1313 1010 1111
September 20, 2004 (c) Piotr Indyk & Charles Leiserson L4.27
Example of partitioning
66 1010 1313 55 88 33 22 1111
66 55 33 1010 88 1313 22 1111
66 55 1313 1010 88 33 22 1111
i j66 55 33 22 88 1313 1010 1111
September 20, 2004 (c) Piotr Indyk & Charles Leiserson L4.28
Example of partitioning
66 1010 1313 55 88 33 22 1111
66 55 33 1010 88 1313 22 1111
66 55 1313 1010 88 33 22 1111
i j66 55 33 22 88 1313 1010 1111
September 20, 2004 (c) Piotr Indyk & Charles Leiserson L4.29
Example of partitioning
66 1010 1313 55 88 33 22 1111
66 55 33 1010 88 1313 22 1111
66 55 1313 1010 88 33 22 1111
66 55 33 22 88 1313 1010 1111
i22 55 33 66 88 1313 1010 1111
September 20, 2004 (c) Piotr Indyk & Charles Leiserson L4.30
xx
Analysis of quicksort
• Assume all input elements are distinct.• In practice, there are better partitioning
algorithms for when duplicate input elements may exist.
• What is the worst case running time of Quicksort ?
≤ x≤ x ≥ x≥ x
September 20, 2004 (c) Piotr Indyk & Charles Leiserson L4.31
Worst-case of quicksort
• Input sorted or reverse sorted.• Partition around min or max element.• One side of partition always has no elements.
)()()1(
)()1()1()()1()0()(
2nnnT
nnTnnTTnT
Θ=
Θ+−=Θ+−+Θ=Θ+−+=
(arithmetic series)
September 20, 2004 (c) Piotr Indyk & Charles Leiserson L4.32
Worst-case recursion treeT(n) = T(0) + T(n–1) + cn
September 20, 2004 (c) Piotr Indyk & Charles Leiserson L4.33
Worst-case recursion treeT(n) = T(0) + T(n–1) + cn
T(n)
September 20, 2004 (c) Piotr Indyk & Charles Leiserson L4.34
cnT(0) T(n–1)
Worst-case recursion treeT(n) = T(0) + T(n–1) + cn
September 20, 2004 (c) Piotr Indyk & Charles Leiserson L4.35
cnT(0) c(n–1)
Worst-case recursion treeT(n) = T(0) + T(n–1) + cn
T(0) T(n–2)
September 20, 2004 (c) Piotr Indyk & Charles Leiserson L4.36
cnT(0) c(n–1)
Worst-case recursion treeT(n) = T(0) + T(n–1) + cn
T(0) c(n–2)
T(0)
Θ(1)
O
September 20, 2004 (c) Piotr Indyk & Charles Leiserson L4.37
cnT(0) c(n–1)
Worst-case recursion treeT(n) = T(0) + T(n–1) + cn
T(0) c(n–2)
T(0)
Θ(1)
O
( )2
1nk
n
kΘ=
Θ ∑
=
September 20, 2004 (c) Piotr Indyk & Charles Leiserson L4.38
cnΘ(1) c(n–1)
Worst-case recursion treeT(n) = T(0) + T(n–1) + cn
Θ(1) c(n–2)
Θ(1)
Θ(1)
O
( )2
1nk
n
kΘ=
Θ ∑
=
h = n
September 20, 2004 (c) Piotr Indyk & Charles Leiserson L4.39
Nice-case analysis
If we’re lucky, PARTITION splits the array evenly:T(n) = 2T(n/2) + Θ(n)
= Θ(n lg n) (same as merge sort)
What if the split is always 109
101 : ?
( ) ( ) )()( 109
101 nnTnTnT Θ++=
September 20, 2004 (c) Piotr Indyk & Charles Leiserson L4.40
Analysis of nice case)(nT
September 20, 2004 (c) Piotr Indyk & Charles Leiserson L4.41
Analysis of nice casecn
( )nT 101 ( )nT 10
9
September 20, 2004 (c) Piotr Indyk & Charles Leiserson L4.42
Analysis of nice casecn
cn101 cn10
9
( )nT 1001 ( )nT 100
9 ( )nT 1009 ( )nT 100
81
September 20, 2004 (c) Piotr Indyk & Charles Leiserson L4.43
Analysis of nice casecn
cn101 cn10
9
cn1001 cn100
9 cn1009 cn100
81
Θ(1)
Θ(1)
… …log10/9n
cn
cn
cn
…
September 20, 2004 (c) Piotr Indyk & Charles Leiserson L4.44
log10n
Analysis of nice casecn
cn101 cn10
9
cn1001 cn100
9 cn1009 cn100
81
Θ(1)
Θ(1)
… …log10/9n
cn
cn
cn
T(n) ≤ cn log10/9n + Ο(n)
…
cn log10n ≤
September 20, 2004 (c) Piotr Indyk & Charles Leiserson L4.45
Randomized quicksort• Partition around a random element. I.e.,
around A[t] , where t chosen uniformly at random from p…r
• We will show that the expected time is O(n log n)
September 20, 2004 (c) Piotr Indyk & Charles Leiserson L4.46
“Paranoid” quicksort
• Will modify the algorithm to make it easier to analyze:
• Repeat:• Choose the pivot to be a random element of the array• Perform PARTITION
• Until the resulting split is “lucky”, i.e., not worse than 1/10: 9/10• Recurse on both sub-arrays
September 20, 2004 (c) Piotr Indyk & Charles Leiserson L4.47
Analysis• Let T(n) be an upper bound on the expectedrunning time on any array of n elements• Consider any input of size n• The time needed to sort the input is bounded from the above by a sum of
• The time needed to sort the left subarray• The time needed to sort the right subarray• The number of iterations until we get a lucky split, times cn
September 20, 2004 (c) Piotr Indyk & Charles Leiserson L4.48
Expectations
cnpartitionsEinTiTnT •+−+≤ ][#)()(max)(
• By linearity of expectation:
where maximum is taken over i ∈ [n/10,9n/10]• We will show that E[#partitions] is ≤ 10/8• Therefore:
]10/9,10/[,2)()(max)( nnicninTiTnT ∈+−+≤
September 20, 2004 (c) Piotr Indyk & Charles Leiserson L4.49
Final bound
• Can use the recursion tree argument:• Tree depth is Θ(log n)• Total expected work at each level is at most 10/8 cn• The total expected time is Ο(n log n)
September 20, 2004 (c) Piotr Indyk & Charles Leiserson L4.50
Lucky partitions
• The probability that a random pivot induces lucky partition is at least 8/10(we are not lucky if the pivot happens to be among the smallest/largest n/10 elements)
• If we flip a coin, with heads prob. p=8/10 , the expected waiting time for the first head is 1/p = 10/8
September 20, 2004 (c) Piotr Indyk & Charles Leiserson L4.51
Quicksort in practice
• Quicksort is a great general-purpose sorting algorithm.
• Quicksort is typically over twice as fast as merge sort.
• Quicksort can benefit substantially from code tuning.
• Quicksort behaves well even with caching and virtual memory.
• Quicksort is great!
September 20, 2004 (c) Piotr Indyk & Charles Leiserson L4.52
More intuitionSuppose we alternate lucky, unlucky, lucky, unlucky, lucky, ….
L(n) = 2U(n/2) + Θ(n) luckyU(n) = L(n – 1) + Θ(n) unlucky
Solving:L(n) = 2(L(n/2 – 1) + Θ(n/2)) + Θ(n)
= 2L(n/2 – 1) + Θ(n)= Θ(n lg n)
How can we make sure we are usually lucky?Lucky!
September 20, 2004 (c) Piotr Indyk & Charles Leiserson L4.53
Randomized quicksort analysis
Let T(n) = the random variable for the running time of randomized quicksort on an input of size n, assuming random numbers are independent.For k = 0, 1, …, n–1, define the indicator random variable
Xk =1 if PARTITION generates a k : n–k–1 split,0 otherwise.
E[Xk] = PrXk = 1 = 1/n, since all splits are equally likely, assuming elements are distinct.
September 20, 2004 (c) Piotr Indyk & Charles Leiserson L4.54
Analysis (continued)
T(n) =
T(0) + T(n–1) + Θ(n) if 0 : n–1 split,T(1) + T(n–2) + Θ(n) if 1 : n–2 split,
MT(n–1) + T(0) + Θ(n) if n–1 : 0 split,
( )∑−
=Θ+−−+=
1
0)()1()(
n
kk nknTkTX .
September 20, 2004 (c) Piotr Indyk & Charles Leiserson L4.55
Calculating expectation( )
Θ+−−+= ∑
−
=
1
0)()1()()]([
n
kk nknTkTXEnTE
Take expectations of both sides.
September 20, 2004 (c) Piotr Indyk & Charles Leiserson L4.56
Calculating expectation( )
( )[ ]∑
∑−
=
−
=
Θ+−−+=
Θ+−−+=
1
0
1
0
)()1()(
)()1()()]([
n
kk
n
kk
nknTkTXE
nknTkTXEnTE
Linearity of expectation.
September 20, 2004 (c) Piotr Indyk & Charles Leiserson L4.57
Calculating expectation( )
( )[ ]
[ ] [ ]∑
∑
∑
−
=
−
=
−
=
Θ+−−+⋅=
Θ+−−+=
Θ+−−+=
1
0
1
0
1
0
)()1()(
)()1()(
)()1()()]([
n
kk
n
kk
n
kk
nknTkTEXE
nknTkTXE
nknTkTXEnTE
Independence of Xk from other random choices.
September 20, 2004 (c) Piotr Indyk & Charles Leiserson L4.58
Calculating expectation( )
( )[ ]
[ ] [ ]
[ ] [ ] ∑∑∑
∑
∑
∑
−
=
−
=
−
=
−
=
−
=
−
=
Θ+−−+=
Θ+−−+⋅=
Θ+−−+=
Θ+−−+=
1
0
1
0
1
0
1
0
1
0
1
0
)(1)1(1)(1
)()1()(
)()1()(
)()1()()]([
n
k
n
k
n
k
n
kk
n
kk
n
kk
nn
knTEn
kTEn
nknTkTEXE
nknTkTXE
nknTkTXEnTE
Linearity of expectation; E[Xk] = 1/n .
September 20, 2004 (c) Piotr Indyk & Charles Leiserson L4.59
Calculating expectation( )
( )[ ]
[ ] [ ]
[ ] [ ]
[ ] )()(2
)(1)1(1)(1
)()1()(
)()1()(
)()1()()]([
1
1
1
0
1
0
1
0
1
0
1
0
1
0
nkTEn
nn
knTEn
kTEn
nknTkTEXE
nknTkTXE
nknTkTXEnTE
n
k
n
k
n
k
n
k
n
kk
n
kk
n
kk
Θ+=
Θ+−−+=
Θ+−−+⋅=
Θ+−−+=
Θ+−−+=
∑
∑∑∑
∑
∑
∑
−
=
−
=
−
=
−
=
−
=
−
=
−
=
Summations have identical terms.
September 20, 2004 (c) Piotr Indyk & Charles Leiserson L4.60
Hairy recurrence
[ ] )()(2)]([1
2nkTE
nnTE
n
kΘ+= ∑
−
=
(The k = 0, 1 terms can be absorbed in the Θ(n).)
Prove: E[T(n)] ≤ an lg n for constant a > 0 .
Use fact: 21
2812
21 lglg nnnkk
n
k∑
−
=−≤ (exercise).
• Choose a large enough so that an lg ndominates E[T(n)] for sufficiently small n ≥ 2.
September 20, 2004 (c) Piotr Indyk & Charles Leiserson L4.61
Substitution method
[ ] )(lg2)(1
2nkak
nnTE
n
kΘ+≤ ∑
−
=
Substitute inductive hypothesis.
September 20, 2004 (c) Piotr Indyk & Charles Leiserson L4.62
Substitution method
[ ]
)(81lg
212
)(lg2)(
22
1
2
nnnnna
nkakn
nTEn
k
Θ+
−≤
Θ+≤ ∑−
=
Use fact.
September 20, 2004 (c) Piotr Indyk & Charles Leiserson L4.63
Substitution method
[ ]
Θ−−=
Θ+
−≤
Θ+≤ ∑−
=
)(4
lg
)(81lg
212
)(lg2)(
22
1
2
nannan
nnnnna
nkakn
nTEn
k
Express as desired – residual.
September 20, 2004 (c) Piotr Indyk & Charles Leiserson L4.64
Substitution method
[ ]
nan
nannan
nnnnna
nkakn
nTEn
k
lg
)(4
lg
)(81lg
212
)(lg2)(
22
1
2
≤
Θ−−=
Θ+
−=
Θ+≤ ∑−
=
if a is chosen large enough so that an/4 dominates the Θ(n).
,
September 20, 2004 (c) Piotr Indyk & Charles Leiserson L4.65
• Assume Running time= O(n) for nelements.
Running time= O(n) for nelements.
September 20, 2004 (c) Piotr Indyk & Charles Leiserson L4.66
Randomized Algorithms
• Algorithms that make decisions based on random coin flips.
• Can “fool” the adversary.• The running time (or even correctness) is a
random variable; we measure the expectedrunning time.
• We assume all random choices are independent .
• This is not the average case !
Introduction to Algorithms6.046J/18.401J
Lecture 5Prof. Piotr Indyk
Introduction to Algorithms September 22, 2004 L5.2© Piotr Indyk and Charles Leiserson
Today
• Order statistics (e.g., finding median)• Two O(n) time algorithms:
– Randomized: similar to Quicksort– Deterministic: quite tricky
• Both are examples of divide and conquer
Introduction to Algorithms September 22, 2004 L5.3© Piotr Indyk and Charles Leiserson
Order statisticsSelect the ith smallest of n elements (the element with rank i).• i = 1: minimum;• i = n: maximum;• i = (n+1)/2 or (n+1)/2: median.
How fast can we solve the problem ?• Min/max: O(n)• General i : O(n log n) by sorting• We will see how to do it in O(n) time
Introduction to Algorithms September 22, 2004 L5.4© Piotr Indyk and Charles Leiserson
Randomized Algorithm for Finding the ith element
• Divide and Conquer Approach• Main idea: PARTITION
≤ x≤ x ≥ x≥ xp q
kx
• If i<k, recurse on the left• If i>k, recurse on the right• Otherwise, output x
Introduction to Algorithms September 22, 2004 L5.5© Piotr Indyk and Charles Leiserson
Randomized Divide-and-Conquer
RAND-SELECT(A, p, r, i)if p = r then return A[p]q ← RAND-PARTITION(A, p, r)k ← q – p + 1 ⊳ k=rank(A[q])if i = k then return A[q]if i < k
then return RAND-SELECT(A, p, q – 1, i )else return RAND-SELECT(A, q + 1, r, i – k )
≤ A[q]≤ A[q] ≥ A[q]≥ A[q]qp r
k
Introduction to Algorithms September 22, 2004 L5.6© Piotr Indyk and Charles Leiserson
Example
pivoti = 766 1010 1313 55 88 33 22 1111
k = 4
Select the 7 – 4 = 3rd smallest recursively.
Select the i = 7th smallest:
22 55 33 66 88 1313 1010 1111Partition:
Introduction to Algorithms September 22, 2004 L5.7© Piotr Indyk and Charles Leiserson
Analysis
Lucky:101log 9/10 == nn
CASE 3T(n) = T(9n/10) + Θ(n)
= Θ(n)
• What is the worst-case running time ?Unlucky:
T(n) = T(n – 1) + Θ(n)= Θ(n2)
arithmetic series
• Recall that a lucky partition splits into arrays with size ratio at most 9:1
• What if all partitions are lucky ?
Introduction to Algorithms September 22, 2004 L5.8© Piotr Indyk and Charles Leiserson
Expected Running Time
• The probability that a random pivot induces lucky partition is at least 8/10 (Lecture 4)
• Let ti be the number of partitions performed between the (i-1) -th and the i-th lucky partition
• The total time is at most…T = t1 n + t2 (9/10) n + t3 (9/10)2 n + …
• The total expected time is at most:E[T]=E[t1] n + E[t2] (9/10) n + E[t3] (9/10)2 n + …
= 10/8 * [n + (9/10)n + …]= O(n)
Introduction to Algorithms September 22, 2004 L5.9© Piotr Indyk and Charles Leiserson
Digression: 9 to 1
• Do we need to define the lucky partition as 9:1 balanced ?
• No. Suffices to say that both sides have size ≥αn , for 0< α <½
• Probability of getting a lucky partition is1-2α
Introduction to Algorithms September 22, 2004 L5.10© Piotr Indyk and Charles Leiserson
How Does it Work In Practice?
• Need 7 volunteers (a.k.a. elements)• Will choose the median according to height
Introduction to Algorithms September 22, 2004 L5.11© Piotr Indyk and Charles Leiserson
xx
Partitioning subroutinePARTITION(A, p, r) ⊳ A[p . . r]
x ← A[p] ⊳ pivot = A[p]i ← pfor j ← p + 1 to r
do if A[ j] ≤ xthen i ← i + 1
exchange A[i] ↔ A[ j]exchange A[p] ↔ A[i]return i
≤ x≤ x ≥ x≥ x ??p i rj
Invariant:
Introduction to Algorithms September 22, 2004 L5.12© Piotr Indyk and Charles Leiserson
Summary of randomized order-statistic selection
• Works fast: linear expected time.• Excellent algorithm in practice.• But, the worst case is very bad: Θ(n2).
Q. Is there an algorithm that runs in linear time in the worst case?
IDEA: Generate a good pivot recursively.
A. Yes, due to [Blum-Floyd-Pratt-Rivest-Tarjan’73].
Introduction to Algorithms September 22, 2004 L5.13© Piotr Indyk and Charles Leiserson
Worst-case linear-time order statistics
if i = k then return xelseif i < k
then recursively SELECT the ith smallest element in the lower part
else recursively SELECT the (i–k)th smallest element in the upper part
SELECT(i, n)1. Divide the n elements into groups of 5. Find
the median of each 5-element group by hand.2. Recursively SELECT the median x of the n/5
group medians to be the pivot.3. Partition around the pivot x. Let k = rank(x).4.
Same as RAND-SELECT
Introduction to Algorithms September 22, 2004 L5.14© Piotr Indyk and Charles Leiserson
Choosing the pivot
Introduction to Algorithms September 22, 2004 L5.15© Piotr Indyk and Charles Leiserson
Choosing the pivot
1. Divide the n elements into groups of 5.
Introduction to Algorithms September 22, 2004 L5.16© Piotr Indyk and Charles Leiserson
Choosing the pivot
lesser
greater
1. Divide the n elements into groups of 5. Find the median of each 5-element group by rote.
Introduction to Algorithms September 22, 2004 L5.17© Piotr Indyk and Charles Leiserson
Choosing the pivot
lesser
greater
1. Divide the n elements into groups of 5. Find the median of each 5-element group by rote.
2. Recursively SELECT the median x of the n/5group medians to be the pivot.
x
Introduction to Algorithms September 22, 2004 L5.18© Piotr Indyk and Charles Leiserson
Analysis
lesser
greater
x
At least half the group medians are ≤ x, which is at least n/5 /2 = n/10 group medians.
Introduction to Algorithms September 22, 2004 L5.19© Piotr Indyk and Charles Leiserson
Analysis
lesser
greater
x
At least half the group medians are ≤ x, which is at least n/5 /2 = n/10 group medians.• Therefore, at least 3 n/10 elements are ≤ x.
Introduction to Algorithms September 22, 2004 L5.20© Piotr Indyk and Charles Leiserson
Analysis
lesser
greater
x
At least half the group medians are ≤ x, which is at least n/5 /2 = n/10 group medians.• Therefore, at least 3 n/10 elements are ≤ x.• Similarly, at least 3 n/10 elements are ≥ x.
Introduction to Algorithms September 22, 2004 L5.21© Piotr Indyk and Charles Leiserson
Developing the recurrence
if i = k then return xelseif i < k
then recursively SELECT the ith smallest element in the lower part
else recursively SELECT the (i–k)th smallest element in the upper part
SELECT(i, n)1. Divide the n elements into groups of 5. Find
the median of each 5-element group by rote.2. Recursively SELECT the median x of the n/5
group medians to be the pivot.3. Partition around the pivot x. Let k = rank(x).4.
T(n)
Θ(n)
T(n/5)Θ(n)
T(7n/10)
Introduction to Algorithms September 22, 2004 L5.22© Piotr Indyk and Charles Leiserson
Solving the recurrence)(
107
51)( nnTnTnT Θ+
+
=
if c is chosen large enough to handle the Θ(n).
cn
ncncn
ncn
ncncnnT
≤
Θ−−=
Θ+=
Θ++≤
)(202
)(2018
)(107
51)(Substitution:
T(n) ≤ cn
Introduction to Algorithms September 22, 2004 L5.23© Piotr Indyk and Charles Leiserson
Minor simplification• For n ≥ 50, we have 3 n/10 ≥ n/4.• Therefore, for n ≥ 50 the recursive call to
SELECT in Step 4 is executed recursively on ≤ 3n/4 elements.
• Thus, the recurrence for running time can assume that Step 4 takes time T(3n/4) in the worst case.
• For n < 50, we know that the worst-case time is T(n) = Θ(1).
Introduction to Algorithms September 22, 2004 L5.24© Piotr Indyk and Charles Leiserson
Conclusions• Since the work at each level of recursion
is a constant fraction (18/20) smaller, the work per level is a geometric series dominated by the linear work at the root.
• In practice, this algorithm runs slowly, because the constant in front of n is large.
• The randomized algorithm is far more practical.
Exercise: Why not divide into groups of 3?
Introduction to Algorithms6.046J/18.401J
Lecture 6Prof. Piotr Indyk
Introduction to Algorithms September 27, 2004 L6.2© Charles E. Leiserson and Piotr Indyk
Today: sorting
• Show that Θ (n lg n) is the best possible running time for a sorting algorithm.
• Design an algorithm that sorts in O(n) time.• Hint: different models ?
Introduction to Algorithms September 27, 2004 L6.3© Charles E. Leiserson and Piotr Indyk
Comparison sortAll the sorting algorithms we have seen so far are comparison sorts: only use comparisons to determine the relative order of elements.• E.g., insertion sort, merge sort, quicksort,
heapsort.
Introduction to Algorithms September 27, 2004 L6.4© Charles E. Leiserson and Piotr Indyk
xx
Partitioning subroutinePARTITION(A, p, r) ⊳ A[p . . r]
x ← A[p] ⊳ pivot = A[p]i ← pfor j ← p + 1 to r
do if A[ j] ≤ xthen i ← i + 1
exchange A[i] ↔ A[ j]exchange A[p] ↔ A[i]return i
≤ x≤ x ≥ x≥ x ??p i rj
Invariant:
Introduction to Algorithms September 27, 2004 L6.5© Charles E. Leiserson and Piotr Indyk
Comparison sort
• All of our algorithms used comparisons• All of our algorithms have running time
Ω(n lg n)• Is it the best that we can do using just
comparisons ?• Answer: YES, via decision trees
Introduction to Algorithms September 27, 2004 L6.6© Charles E. Leiserson and Piotr Indyk
Decision-tree example
1:21:2
2:32:3
123123 1:31:3
132132 312312
1:31:3
213213 2:32:3
231231 321321
Each internal node is labeled i:j for i, j ∈ 1, 2,…, n.•The left subtree shows subsequent comparisons if ai ≤ aj.•The right subtree shows subsequent comparisons if ai ≥ aj.
Sort ⟨a1, a2, …, an⟩(n=3)
Introduction to Algorithms September 27, 2004 L6.7© Charles E. Leiserson and Piotr Indyk
Decision-tree example
1:21:2
2:32:3
123123 1:31:3
132132 312312
1:31:3
213213 2:32:3
231231 321321
Each internal node is labeled i:j for i, j ∈ 1, 2,…, n.•The left subtree shows subsequent comparisons if ai ≤ aj.•The right subtree shows subsequent comparisons if ai ≥ aj.
Sort ⟨a1, a2, a3⟩= ⟨ 9, 4, 6 ⟩:
Introduction to Algorithms September 27, 2004 L6.8© Charles E. Leiserson and Piotr Indyk
Decision-tree example
1:21:2
2:32:3
123123 1:31:3
132132 312312
1:31:3
213213 2:32:3
231231 321321
Each internal node is labeled i:j for i, j ∈ 1, 2,…, n.•The left subtree shows subsequent comparisons if ai ≤ aj.•The right subtree shows subsequent comparisons if ai ≥ aj.
9 ≥ 4Sort ⟨a1, a2, a3⟩= ⟨ 9, 4, 6 ⟩:
Introduction to Algorithms September 27, 2004 L6.9© Charles E. Leiserson and Piotr Indyk
Decision-tree example
1:21:2
2:32:3
123123 1:31:3
132132 312312
1:31:3
213213 2:32:3
231231 321321
Each internal node is labeled i:j for i, j ∈ 1, 2,…, n.•The left subtree shows subsequent comparisons if ai ≤ aj.•The right subtree shows subsequent comparisons if ai ≥ aj.
9 ≥ 6
Sort ⟨a1, a2, a3⟩= ⟨ 9, 4, 6 ⟩:
Introduction to Algorithms September 27, 2004 L6.10© Charles E. Leiserson and Piotr Indyk
Decision-tree example
1:21:2
2:32:3
123123 1:31:3
132132 312312
1:31:3
213213 2:32:3
231231 321321
Each internal node is labeled i:j for i, j ∈ 1, 2,…, n.•The left subtree shows subsequent comparisons if ai ≤ aj.•The right subtree shows subsequent comparisons if ai ≥ aj.
4 ≤ 6
Sort ⟨a1, a2, a3⟩= ⟨ 9, 4, 6 ⟩:
Introduction to Algorithms September 27, 2004 L6.11© Charles E. Leiserson and Piotr Indyk
Decision-tree example
1:21:2
2:32:3
123123 1:31:3
132132 312312
1:31:3
213213 2:32:3
231231 321321
Each leaf contains a permutation ⟨π(1), π(2),…, π(n)⟩ to indicate that the ordering aπ(1) ≤ aπ(2) ≤ L ≤ aπ(n) has been established.
4 ≤ 6 ≤ 9
Sort ⟨a1, a2, a3⟩= ⟨ 9, 4, 6 ⟩:
Introduction to Algorithms September 27, 2004 L6.12© Charles E. Leiserson and Piotr Indyk
Decision-tree modelA decision tree can model the execution of any comparison sort:• One tree for each input size n. • View the algorithm as splitting whenever it compares two
elements.• The tree contains the comparisons along all possible
instruction traces.• The number of comparisons done by the algorithm on a given
input =…the length of the path taken.
• Worst-case number of comparisons =…max path length = height of tree.
• Worst-case time ≥ worst-case number of comparisons
Introduction to Algorithms September 27, 2004 L6.13© Charles E. Leiserson and Piotr Indyk
Lower bound for decision-tree sorting
Theorem. Any decision tree that can sort n elements must have height Ω(n lg n) .
Corollary. Any comparison sorting algorithm has worst-case running time Ω(n lg n).
Corollary 2. Merge sort and Heap Sort areasymptotically optimal comparison sorting algorithms.
Introduction to Algorithms September 27, 2004 L6.14© Charles E. Leiserson and Piotr Indyk
Lower bound for decision-tree sorting
Theorem. Any decision tree that can sort n elements must have height Ω(n lg n) .Proof.• The tree must contain ≥ n! leaves, since there are n! possible permutations• A height-h binary tree has ≤ 2h leaves• Thus, 2h ≥ #leaves ≥ n! , or h ≥ lg(n!)
Introduction to Algorithms September 27, 2004 L6.15© Charles E. Leiserson and Piotr Indyk
Proof, ctd.
2h ≥ n! ≥ n*(n-1)*…* n/2≥ (n/2)n/2
⇒ h ≥ lg( (n/2)n/2 )≥ (n/2) (lg n –lg 2)= Ω(n lg n) .
Introduction to Algorithms September 27, 2004 L6.16© Charles E. Leiserson and Piotr Indyk
Example: sorting 3 elements
Recall h ≥ lg(n!)• n=3• n!=6• log26 = 2.58• Sorting 3 elements requires…
… ≥3 comparisons in the worst case
Introduction to Algorithms September 27, 2004 L6.17© Charles E. Leiserson and Piotr Indyk
Decision-tree for n=3
1:21:2
2:32:3
123123 1:31:3
132132 312312
1:31:3
213213 2:32:3
231231 321321
Sort ⟨a1, a2, a3⟩
Introduction to Algorithms September 27, 2004 L6.18© Charles E. Leiserson and Piotr Indyk
Sorting in linear time
Counting sort: No comparisons between elements.• Input: A[1 . . n], where A[ j]∈1, 2, …, k .• Output: B[1 . . n], sorted*• Auxiliary storage: C[1 . . k] .
*Actually, we require the algorithm to construct a permutation of the inputarray A that produces the sorted array B. This permutation can be obtainedby making small changes to the last loop of the algorithm.
Introduction to Algorithms September 27, 2004 L6.19© Charles E. Leiserson and Piotr Indyk
Counting sort
for i ← 1 to kdo C[i] ← 0
for j ← 1 to ndo C[A[ j]] ← C[A[ j]] + 1 ⊳ C[i] = |key = i|
for i ← 2 to kdo C[i] ← C[i] + C[i–1] ⊳ C[i] = |key ≤ i|
for j ← n downto 1do B[C[A[ j]]] ← A[ j]
C[A[ j]] ← C[A[ j]] – 1
Introduction to Algorithms September 27, 2004 L6.20© Charles E. Leiserson and Piotr Indyk
Counting-sort example
A: 44 11 33 44 33
B:
1 2 3 4 5
C:1 2 3 4
Introduction to Algorithms September 27, 2004 L6.21© Charles E. Leiserson and Piotr Indyk
Loop 1
A: 44 11 33 44 33
B:
1 2 3 4 5
C: 00 00 00 001 2 3 4
for i ← 1 to kdo C[i] ← 0
Introduction to Algorithms September 27, 2004 L6.22© Charles E. Leiserson and Piotr Indyk
Loop 2
A: 44 11 33 44 33
B:
1 2 3 4 5
C: 00 00 00 111 2 3 4
for j ← 1 to ndo C[A[ j]] ← C[A[ j]] + 1 ⊳ C[i] = |key = i|
Introduction to Algorithms September 27, 2004 L6.23© Charles E. Leiserson and Piotr Indyk
Loop 2
A: 44 11 33 44 33
B:
1 2 3 4 5
C: 11 00 00 111 2 3 4
for j ← 1 to ndo C[A[ j]] ← C[A[ j]] + 1 ⊳ C[i] = |key = i|
Introduction to Algorithms September 27, 2004 L6.24© Charles E. Leiserson and Piotr Indyk
Loop 2
A: 44 11 33 44 33
B:
1 2 3 4 5
C: 11 00 11 111 2 3 4
for j ← 1 to ndo C[A[ j]] ← C[A[ j]] + 1 ⊳ C[i] = |key = i|
Introduction to Algorithms September 27, 2004 L6.25© Charles E. Leiserson and Piotr Indyk
Loop 2
A: 44 11 33 44 33
B:
1 2 3 4 5
C: 11 00 11 221 2 3 4
for j ← 1 to ndo C[A[ j]] ← C[A[ j]] + 1 ⊳ C[i] = |key = i|
Introduction to Algorithms September 27, 2004 L6.26© Charles E. Leiserson and Piotr Indyk
Loop 2
A: 44 11 33 44 33
B:
1 2 3 4 5
C: 11 00 22 221 2 3 4
for j ← 1 to ndo C[A[ j]] ← C[A[ j]] + 1 ⊳ C[i] = |key = i|
Introduction to Algorithms September 27, 2004 L6.27© Charles E. Leiserson and Piotr Indyk
Loop 3
A: 44 11 33 44 33
B:
1 2 3 4 5
C: 11 00 22 221 2 3 4
C': 11 11 22 22
for i ← 2 to kdo C[i] ← C[i] + C[i–1] ⊳ C[i] = |key ≤ i|
Introduction to Algorithms September 27, 2004 L6.28© Charles E. Leiserson and Piotr Indyk
Loop 3
A: 44 11 33 44 33
B:
1 2 3 4 5
C: 11 00 22 221 2 3 4
C': 11 11 33 22
for i ← 2 to kdo C[i] ← C[i] + C[i–1] ⊳ C[i] = |key ≤ i|
Introduction to Algorithms September 27, 2004 L6.29© Charles E. Leiserson and Piotr Indyk
Loop 3
A: 44 11 33 44 33
B:
1 2 3 4 5
C: 11 00 22 221 2 3 4
C': 11 11 33 55
for i ← 2 to kdo C[i] ← C[i] + C[i–1] ⊳ C[i] = |key ≤ i|
Introduction to Algorithms September 27, 2004 L6.30© Charles E. Leiserson and Piotr Indyk
Loop 4
A: 44 11 33 44 33
B:
1 2 3 4 5
C': 11 11 33 55
for j ← n downto 1do B[C[A[ j]]] ← A[ j]
C[A[ j]] ← C[A[ j]] – 1
1 2 3 4
Introduction to Algorithms September 27, 2004 L6.31© Charles E. Leiserson and Piotr Indyk
Loop 4
A: 44 11 33 44 33
B: 33
1 2 3 4 5
C': 11 11 33 55
for j ← n downto 1do B[C[A[ j]]] ← A[ j]
C[A[ j]] ← C[A[ j]] – 1
1 2 3 4
Introduction to Algorithms September 27, 2004 L6.32© Charles E. Leiserson and Piotr Indyk
Loop 4
A: 44 11 33 44 33
B: 33
1 2 3 4 5
C': 11 11 22 55
for j ← n downto 1do B[C[A[ j]]] ← A[ j]
C[A[ j]] ← C[A[ j]] – 1
1 2 3 4
Introduction to Algorithms September 27, 2004 L6.33© Charles E. Leiserson and Piotr Indyk
Loop 4
A: 44 11 33 44 33
B: 33 44
1 2 3 4 5
C': 11 11 22 55
for j ← n downto 1do B[C[A[ j]]] ← A[ j]
C[A[ j]] ← C[A[ j]] – 1
1 2 3 4
Introduction to Algorithms September 27, 2004 L6.34© Charles E. Leiserson and Piotr Indyk
Loop 4
A: 44 11 33 44 33
B: 33 44
1 2 3 4 5
C': 11 11 22 44
for j ← n downto 1do B[C[A[ j]]] ← A[ j]
C[A[ j]] ← C[A[ j]] – 1
1 2 3 4
Introduction to Algorithms September 27, 2004 L6.35© Charles E. Leiserson and Piotr Indyk
Loop 4
A: 44 11 33 44 33
B: 33 33 44
1 2 3 4 5
C': 11 11 22 44
for j ← n downto 1do B[C[A[ j]]] ← A[ j]
C[A[ j]] ← C[A[ j]] – 1
1 2 3 4
Introduction to Algorithms September 27, 2004 L6.36© Charles E. Leiserson and Piotr Indyk
Loop 4
A: 44 11 33 44 33
B: 33 33 44
1 2 3 4 5
C': 11 11 11 44
for j ← n downto 1do B[C[A[ j]]] ← A[ j]
C[A[ j]] ← C[A[ j]] – 1
1 2 3 4
Introduction to Algorithms September 27, 2004 L6.37© Charles E. Leiserson and Piotr Indyk
Loop 4
A: 44 11 33 44 33
B: 11 33 33 44
1 2 3 4 5
C': 11 11 11 44
for j ← n downto 1do B[C[A[ j]]] ← A[ j]
C[A[ j]] ← C[A[ j]] – 1
1 2 3 4
Introduction to Algorithms September 27, 2004 L6.38© Charles E. Leiserson and Piotr Indyk
Loop 4
A: 44 11 33 44 33
B: 11 33 33 44
1 2 3 4 5
C': 00 11 11 44
for j ← n downto 1do B[C[A[ j]]] ← A[ j]
C[A[ j]] ← C[A[ j]] – 1
1 2 3 4
Introduction to Algorithms September 27, 2004 L6.39© Charles E. Leiserson and Piotr Indyk
Loop 4
A: 44 11 33 44 33
B: 11 33 33 44 44
1 2 3 4 5
C': 00 11 11 44
for j ← n downto 1do B[C[A[ j]]] ← A[ j]
C[A[ j]] ← C[A[ j]] – 1
1 2 3 4
Introduction to Algorithms September 27, 2004 L6.40© Charles E. Leiserson and Piotr Indyk
Loop 4
A: 44 11 33 44 33
B: 11 33 33 44 44
1 2 3 4 5
C': 00 11 11 33
for j ← n downto 1do B[C[A[ j]]] ← A[ j]
C[A[ j]] ← C[A[ j]] – 1
1 2 3 4
Introduction to Algorithms September 27, 2004 L6.41© Charles E. Leiserson and Piotr Indyk
B vs C
B: 11 33 33 44 44 C': 11 11 33 55
In the end, each element i occupies the rangeB[C[i-1]+1 … C[i]]
1 2 3 41 2 3 4 5
Introduction to Algorithms September 27, 2004 L6.42© Charles E. Leiserson and Piotr Indyk
Analysisfor i ← 1 to k
do C[i] ← 0
Θ(n)
Θ(k)
Θ(n)
Θ(k)
for j ← 1 to ndo C[A[ j]] ← C[A[ j]] + 1
for i ← 2 to kdo C[i] ← C[i] + C[i–1]
for j ← n downto 1do B[C[A[ j]]] ← A[ j]
C[A[ j]] ← C[A[ j]] – 1Θ(n + k)
Introduction to Algorithms September 27, 2004 L6.43© Charles E. Leiserson and Piotr Indyk
Running time
If k = O(n), then counting sort takes Θ(n) time.• But, sorting takes Ω(n lg n) time!• Why ?
Answer:• Comparison sorting takes Ω(n lg n) time.• Counting sort is not a comparison sort.• In fact, not a single comparison between
elements occurs!
Introduction to Algorithms September 27, 2004 L6.44© Charles E. Leiserson and Piotr Indyk
Stable sorting
Counting sort is a stable sort: it preserves the input order among equal elements.
A: 44 11 33 44 33
B: 11 33 33 44 44
Introduction to Algorithms September 27, 2004 L6.45© Charles E. Leiserson and Piotr Indyk
Sorting integers
• We can sort n integers from 1, 2, …, k in O(n+k) time
• This is nice if k=O(n)• What if, say, k=n2 ?
Introduction to Algorithms September 27, 2004 L6.46© Charles E. Leiserson and Piotr Indyk
Radix sort
• Origin: Herman Hollerith’s card-sorting machine for the 1890 U.S. Census. (See Appendix .)
• Digit-by-digit sort.• Hollerith’s original (bad) idea: sort on
most-significant digit first.• Good idea: Sort on least-significant digit
first with auxiliary stable sort.
Introduction to Algorithms September 27, 2004 L6.47© Charles E. Leiserson and Piotr Indyk
Operation of radix sort
3 2 94 5 76 5 78 3 94 3 67 2 03 5 5
7 2 03 5 54 3 64 5 76 5 73 2 98 3 9
7 2 03 2 94 3 68 3 93 5 54 5 76 5 7
3 2 93 5 54 3 64 5 76 5 77 2 08 3 9
Introduction to Algorithms September 27, 2004 L6.48© Charles E. Leiserson and Piotr Indyk
• Sort on digit t
Correctness of radix sortInduction on digit position • Assume that the numbers
are sorted by their low-order t – 1 digits.
7 2 03 2 94 3 68 3 93 5 54 5 76 5 7
3 2 93 5 54 3 64 5 76 5 77 2 08 3 9
Introduction to Algorithms September 27, 2004 L6.49© Charles E. Leiserson and Piotr Indyk
• Sort on digit t
Correctness of radix sortInduction on digit position • Assume that the numbers
are sorted by their low-order t – 1 digits.
7 2 03 2 94 3 68 3 93 5 54 5 76 5 7
3 2 93 5 54 3 64 5 76 5 77 2 08 3 9
Two numbers that differ in digit t are correctly sorted.
Introduction to Algorithms September 27, 2004 L6.50© Charles E. Leiserson and Piotr Indyk
• Sort on digit t
Correctness of radix sortInduction on digit position • Assume that the numbers
are sorted by their low-order t – 1 digits.
7 2 03 2 94 3 68 3 93 5 54 5 76 5 7
3 2 93 5 54 3 64 5 76 5 77 2 08 3 9
Two numbers that differ in digit t are correctly sorted.Two numbers equal in digit tare put in the same order as the input ⇒ correct order.
Introduction to Algorithms September 27, 2004 L6.51© Charles E. Leiserson and Piotr Indyk
Analysis of radix sort
• Assume counting sort is the auxiliary stable sort.• Sort n computer words of b bits eachE.g., if we sort elements in 1…n2 , b=2 lg n
• Each word can be viewed as having b/r base-2r
digits.
Example: 32-bit word8 8 8 8
r = 8 ⇒ b/r = 4 passes of counting sort on base-28 digits; or r = 16 ⇒ b/r = 2 passes of counting sort on base-216
digits.
Introduction to Algorithms September 27, 2004 L6.52© Charles E. Leiserson and Piotr Indyk
Analysis (continued)Recall: Counting sort takes Θ(n + k) time to sort n numbers in the range from 0 to k – 1.If each b-bit word is broken into r-bit pieces, each pass of counting sort takes Θ(n + 2r) time. Since there are b/r passes, we have
( )
+Θ= rn
rbbnT 2),( .
Choose r to minimize T(n, b):• Increasing r means fewer passes, but as
r > lg n, the time grows exponentially.>
Introduction to Algorithms September 27, 2004 L6.53© Charles E. Leiserson and Piotr Indyk
Choosing r( )
+Θ= rn
rbbnT 2),(
Minimize T(n, b) by differentiating and setting to 0.Or, just observe that we don’t want 2r > n, and there’s no harm asymptotically in choosing r as large as possible subject to this constraint.
>
Choosing r = lg n implies T(n, b) = Θ(bn/lg n) .
• For numbers in the range from 0 to nd – 1, we have b = d lg n ⇒ radix sort runs in Θ(d n) time.
Introduction to Algorithms September 27, 2004 L6.54© Charles E. Leiserson and Piotr Indyk
Conclusions
Example (32-bit numbers):• At most 3 passes when sorting ≥ 2000 numbers.• Merge sort and quicksort do at least lg 2000 =
11 passes.
In practice, radix sort is fast for large inputs, as well as simple to code and maintain.
Downside: Unlike quicksort, radix sort displays little locality of reference.
Introduction to Algorithms September 27, 2004 L6.55© Charles E. Leiserson and Piotr Indyk
Appendix: Punched-card technology
• Herman Hollerith (1860-1929)• Punched cards• Hollerith’s tabulating system• Operation of the sorter• Origin of radix sort• “Modern” IBM card• Web resources on punched-
card technologyReturn to last slide viewed.
Introduction to Algorithms September 27, 2004 L6.56© Charles E. Leiserson and Piotr Indyk
Herman Hollerith(1860-1929)
• The 1880 U.S. Census took almost10 years to process.
• While a lecturer at MIT, Hollerith prototyped punched-card technology.
• His machines, including a “card sorter,” allowed the 1890 census total to be reported in 6 weeks.
• He founded the Tabulating Machine Company in 1911, which merged with other companies in 1924 to form International Business Machines.
Image removed due to copyright considerations.
Introduction to Algorithms September 27, 2004 L6.57© Charles E. Leiserson and Piotr Indyk
Punched cards• Punched card = data record.• Hole = value. • Algorithm = machine + human operator.
Image removed due to copyright considerations.
Introduction to Algorithms September 27, 2004 L6.58© Charles E. Leiserson and Piotr Indyk
Hollerith’s tabulating system•Pantograph card punch
•Hand-press reader•Dial counters•Sorting box
Image removed due to copyright considerations.
Introduction to Algorithms September 27, 2004 L6.59© Charles E. Leiserson and Piotr Indyk
Operation of the sorter• An operator inserts a card into
the press.• Pins on the press reach through
the punched holes to make electrical contact with mercury-filled cups beneath the card.
• Whenever a particular digit value is punched, the lid of the corresponding sorting bin lifts.
• The operator deposits the card into the bin and closes the lid.
• When all cards have been processed, the front panel is opened, and the cards are collected in order, yielding one pass of a stable sort.
Image removed due to copyright considerations.
Introduction to Algorithms September 27, 2004 L6.60© Charles E. Leiserson and Piotr Indyk
Origin of radix sort
Hollerith’s original 1889 patent alludes to a most-significant-digit-first radix sort:
“The most complicated combinations can readily be counted with comparatively few counters or relays by first assorting the cards according to the first items entering into the combinations, then reassorting each group according to the second item entering into the combination, and so on, and finally counting on a few counters the last item of the combination for each group of cards.”
Least-significant-digit-first radix sort seems to be a folk invention originated by machine operators.
Introduction to Algorithms September 27, 2004 L6.61© Charles E. Leiserson and Piotr Indyk
“Modern” IBM card
So, that’s why text windows have 80 columns!
• One character per column.
Image removed due to copyright considerations.
Introduction to Algorithms September 27, 2004 L6.62© Charles E. Leiserson and Piotr Indyk
Web resources on punched-card technology
• Doug Jones’s punched card index• Biography of Herman Hollerith• The 1890 U.S. Census• Early history of IBM• Pictures of Hollerith’s inventions• Hollerith’s patent application (borrowed
from Gordon Bell’s CyberMuseum)• Impact of punched cards on U.S. history
Introduction to Algorithms6.046J/18.401J
Lecture 7Prof. Piotr Indyk
Introduction to Algorithms September 29, 2004 L7.2© Charles Leiserson and Piotr Indyk
Data Structures
• Role of data structures:– Encapsulate data– Support certain operations (e.g., INSERT,
DELETE, SEARCH)• What data structures do we know already ?• Yes, heap:
– INSERT(x)– DELETE-MIN
Introduction to Algorithms September 29, 2004 L7.3© Charles Leiserson and Piotr Indyk
Dictionary problem
Dictionary T holding n records:
key[x]key[x]record
x
Other fields containing satellite data
Operations on T:• INSERT(T, x)• DELETE(T, x)• SEARCH(T, k)
How should the data structure T be organized?
Introduction to Algorithms September 29, 2004 L7.4© Charles Leiserson and Piotr Indyk
Assumptions
Assumptions:
• The set of keys is K ⊆ U = 0, 1, …, u–1• Keys are distinct
What can we do ?
Introduction to Algorithms September 29, 2004 L7.5© Charles Leiserson and Piotr Indyk
Direct access table
• Create a table T[0…u-1]:
• Benefit:– Each operation takes constant time
• Drawbacks:– The range of keys can be large:
• 64-bit numbers (which represent 18,446,744,073,709,551,616 different keys),
• character strings (even larger!)
T[k] = x if k ∈ K and key[x] = k,NIL otherwise.
Introduction to Algorithms September 29, 2004 L7.6© Charles Leiserson and Piotr Indyk
As each key is inserted, h maps it to a slot of T.
Hash functionsSolution: Use a hash function h to map the universe U of all keys into0, 1, …, m–1:
U
Kk1
k2 k3
k4
k5
0
m–1
h(k1)h(k4)
h(k2)
h(k3)
When a record to be inserted maps to an already occupied slot in T, a collision occurs.
T
= h(k5)
Introduction to Algorithms September 29, 2004 L7.7© Charles Leiserson and Piotr Indyk
Collisions resolution by chaining
• Records in the same slot are linked into a list.
h(49) = h(86) = h(52) = i
T
4949 8686 5252i
Introduction to Algorithms September 29, 2004 L7.8© Charles Leiserson and Piotr Indyk
Hash functions
• Designing good functions is quite non-trivial
• For now, we assume they exist. Namely, we assume simple uniform hashing:– Each key k ∈ K of keys is equally likely
to be hashed to any slot of table T, independent of where other keys are hashed
Introduction to Algorithms September 29, 2004 L7.9© Charles Leiserson and Piotr Indyk
Analysis of chaining
Let n be the number of keys in the table, and let m be the number of slots.Define the load factor of T to be
α = n/m= average number of keys per slot.
Introduction to Algorithms September 29, 2004 L7.10© Charles Leiserson and Piotr Indyk
Search cost
Expected time to search for a record with a given key =
apply hash function and access slot
search the list
Expected search time = Θ(1) if α = O(1), or equivalently, if n = O(m).
Θ(1 + α).
Introduction to Algorithms September 29, 2004 L7.11© Charles Leiserson and Piotr Indyk
Other operations
• Insertion time ?– Constant: hash and add to the list
• Deletion time ? Recall that we defined DELETE(T, x)– Also constant, if x has a pointer to the
collision list and list doubly linked– Otherwise, do SEARCH first
Introduction to Algorithms September 29, 2004 L7.12© Charles Leiserson and Piotr Indyk
Delete
T
4949 8686 5252i
key[x]key[x]
Introduction to Algorithms September 29, 2004 L7.13© Charles Leiserson and Piotr Indyk
Dealing with wishful thinking
The assumption of simple uniform hashing is hard to guarantee, but several common techniques tend to work well in practice as long as their deficiencies can be avoided.
Desirata:• A good hash function should distribute the
keys uniformly into the slots of the table.• Regularity in the key distribution (e.g.,
arithmetic progression) should not affect this uniformity.
Introduction to Algorithms September 29, 2004 L7.14© Charles Leiserson and Piotr Indyk
Hashing in practice
Leaving the realm of Provable
Introduction to Algorithms September 29, 2004 L7.15© Charles Leiserson and Piotr Indyk
h(k)
Division methodDefine
h(k) = k mod m.
Extreme deficiency: If m = 2r, then the hash doesn’t even depend on all the bits of k:• If k = 10110001110110102 and r = 6, then
h(k) = 0110102 .
Deficiency: Don’t pick an m that has a small divisor d. A preponderance of keys that are congruent modulo d can adversely affect uniformity.
Introduction to Algorithms September 29, 2004 L7.16© Charles Leiserson and Piotr Indyk
Division method (continued)
h(k) = k mod m.
Pick m to be a prime.
Annoyance:• Sometimes, making the table size a prime is
inconvenient.But, this method is popular, although the next method we’ll see is usually superior.
Introduction to Algorithms September 29, 2004 L7.17© Charles Leiserson and Piotr Indyk
Multiplication method
Assume that all keys are integers, m = 2r, and our computer has w-bit words. Define
h(k) = (A·k mod 2w) rsh (w – r),where rsh is the “bit-wise right-shift” operator and A is an odd integer in the range 2w–1 < A < 2w.• Don’t pick A too close to 2w.• Multiplication modulo 2w is fast.• The rsh operator is fast.
Introduction to Algorithms September 29, 2004 L7.18© Charles Leiserson and Piotr Indyk
4
0
3526
17
Modular wheel
Multiplication method example
h(k) = (A·k mod 2w) rsh (w – r)Suppose that m = 8 = 23 and that our computer has w = 7-bit words:
1 0 1 1 0 0 1× 1 1 0 1 0 1 1
1 0 0 1 0 1 0 0 1 1 0 0 1 1
= A= k
h(k) A ..2A
..
3A..
Introduction to Algorithms September 29, 2004 L7.19© Charles Leiserson and Piotr Indyk
Back to the realm of Provable
Introduction to Algorithms September 29, 2004 L7.20© Charles Leiserson and Piotr Indyk
A weakness of hashing “as we saw it”
Problem: For any hash function h, a set of keys exists that can cause the average access time of a hash table to skyrocket.• An adversary can pick all keys from
h-1(i) =k ∈ U : h(k) = i for a slot i.
• There is a slot i for which |h-1(i)| ≥u/m
Introduction to Algorithms September 29, 2004 L7.21© Charles Leiserson and Piotr Indyk
Solution
• Randomize!• Choose the hash function at random from
some family of function, and independently of the keys.
• Even if an adversary can see your code, he or she cannot find a bad set of keys, since he or she doesn’t know exactly which hash function will be chosen.
• What family of functions should we select ?
Introduction to Algorithms September 29, 2004 L7.22© Charles Leiserson and Piotr Indyk
Family of hash functions
• Idea #1: Take the family of all functions h: U → 0…m-1
That is, choose each of h(0), h(1), … , h(u-1)independently at random from 0…m-1
• Benefit:– The uniform hashing assumption is true!
• Drawback:– We need u random numbers to specify h.
Where to store them ?
Introduction to Algorithms September 29, 2004 L7.23© Charles Leiserson and Piotr Indyk
Universal hashingIdea #2: Universal Hashing
• Let H be a finite collection of hash functions, each mapping U to 0, 1, …, m–1
• We say H is universal if for all x, y ∈ U, where x ≠ y, we have
Prh∈Hh(x) = h(y)| = 1/m.
Introduction to Algorithms September 29, 2004 L7.24© Charles Leiserson and Piotr Indyk
Universality is good
Theorem. Let h be a hash function chosen (uniformly) at random from a universal set Hof hash functions. Suppose h is used to hash n arbitrary keys into the m slots of a table T. Then, for a given key x, we have
E[#collisions with x] < n/m.
Introduction to Algorithms September 29, 2004 L7.25© Charles Leiserson and Piotr Indyk
Proof of theorem
Proof. Let Cx be the random variable denoting the total number of collisions of keys in T with x, and let
cxy = 1 if h(x) = h(y),0 otherwise.
Note: E[cxy] = 1/m and ∑−∈
=xTyxyx cC .
Introduction to Algorithms September 29, 2004 L7.26© Charles Leiserson and Piotr Indyk
Proof (continued)
= ∑
−∈ ][
xTyxyx cECE • Take expectation
of both sides.
Introduction to Algorithms September 29, 2004 L7.27© Charles Leiserson and Piotr Indyk
Proof (continued)
∑
∑
−∈
−∈
=
=
][
][
xTyxy
xTyxyx
cE
cECE
• Linearity of expectation.
• Take expectation of both sides.
Introduction to Algorithms September 29, 2004 L7.28© Charles Leiserson and Piotr Indyk
Proof (continued)
∑
∑
∑
−∈
−∈
−∈
=
=
=
/1
][
][
xTy
xTyxy
xTyxyx
m
cE
cECE
• E[cxy] = 1/m.
• Linearity of expectation.
• Take expectation of both sides.
Introduction to Algorithms September 29, 2004 L7.29© Charles Leiserson and Piotr Indyk
Proof (continued)
mn
m
cE
cECE
xTy
xTyxy
xTyxyx
1
/1
][
][
−=
=
=
=
∑
∑
∑
−∈
−∈
−∈• Take expectation
of both sides.
• Linearity of expectation.
• E[cxy] = 1/m.
• Algebra..
Introduction to Algorithms September 29, 2004 L7.30© Charles Leiserson and Piotr Indyk
Constructing a set of universal hash functions
• Let m be prime. • Decompose key k into r + 1 digits, each with value in the set 0, 1, …, m–1. • That is, let k = ⟨k0, k1, …, kr⟩, where 0 ≤ ki < m.Randomized strategy:• Pick a = ⟨a0, a1, …, ar⟩ where each ai is chosen randomly from 0, 1, …, m–1.
mkakhr
iiia mod)(
0∑=
=• Define
• Denote H=ha: a as above
Introduction to Algorithms September 29, 2004 L7.31© Charles Leiserson and Piotr Indyk
Universality of dot-product hash functions
Theorem. The set H = ha is universal.
Proof. Suppose that x = ⟨x0, x1, …, xr⟩ and y = ⟨y0, y1, …, yr⟩ are distinct keys. Thus,
they differ in at least one digit position, wlogposition 0. What is the probability that x and ycollide, that is ha(x)=hb(y) ?
Introduction to Algorithms September 29, 2004 L7.32© Charles Leiserson and Piotr Indyk
Proof (continued)
)(mod0)(0
myxar
iiii ≡−∑
=
)(mod0)()(1
000 myxayxar
iiii ≡−+− ∑
=
)(mod)()(1
000 myxayxar
iiii∑
=−−≡− .
)(mod)()(00
myaxabhxhr
iii
r
iiiaa ∑∑
==
≡⇔=
Introduction to Algorithms September 29, 2004 L7.33© Charles Leiserson and Piotr Indyk
Recall PS 2
Theorem. Let m be prime. For any z ∈ Zmsuch that z ≠ 0, there exists a unique z–1 ∈ Zmsuch that
z · z–1 ≡ 1 (mod m).
Introduction to Algorithms September 29, 2004 L7.34© Charles Leiserson and Piotr Indyk
Back to the proof
)(mod)()(1
000 myxayxar
iiii∑
=−−≡−
We have
and since x0 ≠ y0 , an inverse (x0 – y0 )–1 must exist, which implies that
,
)(mod)()( 100
10 myxyxaa
r
iiii
−
=−⋅
−−≡ ∑ .
Thus, for any choices of a1, a2, …, ar, exactly one choice of a0 causes x and y to collide.
Introduction to Algorithms September 29, 2004 L7.35© Charles Leiserson and Piotr Indyk
Proof (completed)
Q. What is the probability that x and ycollide?
A. There are m choices for a0, but exactly one choice for a0 causes x and y to collide, namely
myxyxaar
iiii mod)()( 1
001
0
−⋅
−−= −
=∑ .
Thus, the probability of x and y colliding is 1/m.
Introduction to Algorithms September 29, 2004 L7.36© Charles Leiserson and Piotr Indyk
Recap
• Showed how to implement dictionary so that INSERT, DELETE, SEARCH work in expected constant time under the uniform hashing assumption
• Relaxed the assumption to universal hashing
• Constructed universal hashing for keys in 0…mr -1
Introduction to Algorithms September 29, 2004 L7.37© Charles Leiserson and Piotr Indyk
Perfect hashingGiven a set of n keys, construct a static hash table of size m = O(n) such that SEARCH takes Θ(1) time in the worst case.
IDEA: Two-level scheme with universal hashing at both levels.No collisions at level 2! 4040 3737 2222
0123456
2626
m a 0 1 2 3 4 5 6 7 8
1414 2727
S4
S6
S1
44 3131
11 0000
99 8686
T
h31(14) = h31(27) = 1
Introduction to Algorithms September 29, 2004 L7.38© Charles Leiserson and Piotr Indyk
Collisions at level 2Theorem. Let H be a class of universal hash functions for a table of size m = n2. Then, if we use a random h ∈H to hash n keys into the table, the expected number of collisions is at most 1/2. Proof. By the definition of universality, the probability that 2 given keys in the table collide under h is 1/m = 1/n2. Since there are pairs of keys that can possibly collide, the expected number of collisions is
( )2n
211
2)1(1
2 22 <⋅−=⋅
n
nnn
n .
Introduction to Algorithms September 29, 2004 L7.39© Charles Leiserson and Piotr Indyk
No collisions at level 2Corollary. The probability of no collisions is at least 1/2.
Thus, just by testing random hash functions in H, we’ll quickly find one that works.
Proof. Markov’s inequality says that for any nonnegative random variable X, we have
PrX ≥ t ≤ E[X]/t.Applying this inequality with t = 1, we find that the probability of 1 or more collisions is at most 1/2.
Introduction to Algorithms September 29, 2004 L7.40© Charles Leiserson and Piotr Indyk
Analysis of storageFor the level-1 hash table T, choose m = n, andlet ni be random variable for the number of keys that hash to slot i in T. By using ni
2 slots for the level-2 hash table Si, the expected total storage required for the two-level scheme is therefore
( ) )(1
0
2 nnEm
ii Θ=
Θ∑
−
=,
since the analysis is identical to the analysis from recitation of the expected running time of bucket sort. (For a probability bound, apply Markov.)
Introduction to Algorithms September 29, 2004 L7.41© Charles Leiserson and Piotr Indyk
Resolving collisions by open addressing
No storage is used outside of the hash table itself.• Insertion systematically probes the table until an
empty slot is found.• The hash function depends on both the key and
probe number:h : U × 0, 1, …, m–1 → 0, 1, …, m–1.
• The probe sequence ⟨h(k,0), h(k,1), …, h(k,m–1)⟩should be a permutation of 0, 1, …, m–1.
• The table may fill up, and deletion is difficult (but not impossible).
Introduction to Algorithms September 29, 2004 L7.42© Charles Leiserson and Piotr Indyk
204204
Example of open addressing
Insert key k = 496:
0. Probe h(496,0)586133
481
T0
m–1
collision
Introduction to Algorithms September 29, 2004 L7.43© Charles Leiserson and Piotr Indyk
Example of open addressing
Insert key k = 496:
0. Probe h(496,0)586133
204
481
T0
m–1
1. Probe h(496,1) collision586
Introduction to Algorithms September 29, 2004 L7.44© Charles Leiserson and Piotr Indyk
Example of open addressing
Insert key k = 496:
0. Probe h(496,0)586133
204
481
T0
m–1
1. Probe h(496,1)
insertion496
2. Probe h(496,2)
Introduction to Algorithms September 29, 2004 L7.45© Charles Leiserson and Piotr Indyk
Example of open addressing
Search for key k = 496:
0. Probe h(496,0)586133
204
481
T0
m–1
1. Probe h(496,1)
496
2. Probe h(496,2)
Search uses the same probesequence, terminating suc-cessfully if it finds the keyand unsuccessfully if it encounters an empty slot.
Introduction to Algorithms September 29, 2004 L7.46© Charles Leiserson and Piotr Indyk
Probing strategies
Linear probing:Given an ordinary hash function h′(k), linear probing uses the hash function
h(k,i) = (h′(k) + i) mod m.This method, though simple, suffers from primary clustering, where long runs of occupied slots build up, increasing the average search time. Moreover, the long runs of occupied slots tend to get longer.
Introduction to Algorithms September 29, 2004 L7.47© Charles Leiserson and Piotr Indyk
Probing strategies
Double hashingGiven two ordinary hash functions h1(k) and h2(k), double hashing uses the hash function
h(k,i) = (h1(k) + i⋅h2(k)) mod m.This method generally produces excellent results, but h2(k) must be relatively prime to m. One way is to make m a power of 2 and design h2(k) to produce only odd numbers.
Introduction to Algorithms September 29, 2004 L7.48© Charles Leiserson and Piotr Indyk
Analysis of open addressing
We make the assumption of uniform hashing:• Each key is equally likely to have any one of
the m! permutations as its probe sequence.
Theorem. Given an open-addressed hash table with load factor α = n/m < 1, the expected number of probes in an unsuccessful search is at most 1/(1–α).
Introduction to Algorithms September 29, 2004 L7.49© Charles Leiserson and Piotr Indyk
Proof of the theoremProof.• At least one probe is always necessary.• With probability n/m, the first probe hits an
occupied slot, and a second probe is necessary.• With probability (n–1)/(m–1), the second probe
hits an occupied slot, and a third probe is necessary.
• With probability (n–2)/(m–2), the third probe hits an occupied slot, etc.
Observe that α=<−−
mn
imin for i = 1, 2, …, n.
Introduction to Algorithms September 29, 2004 L7.50© Charles Leiserson and Piotr Indyk
Proof (continued)
Therefore, the expected number of probes is
+−+
−−+
−−++ LL
111
221
1111
nmmn
mn
mn
( )( )( )( )
α
α
ααααααα
−=
=
++++≤++++≤
∑∞
=
11
11111
0
32
i
i
L
LL
.
The textbook has a more rigorous proof.
Introduction to Algorithms September 29, 2004 L7.51© Charles Leiserson and Piotr Indyk
Implications of the theorem
• If α is constant, then accessing an open-addressed hash table takes constant time.
• If the table is half full, then the expected number of probes is 1/(1–0.5) = 2.
• If the table is 90% full, then the expected number of probes is 1/(1–0.9) = 10.
Introduction to Algorithms September 29, 2004 L7.52© Charles Leiserson and Piotr Indyk
Dot-product methodRandomized strategy:Let m be prime. Decompose key k into r + 1digits, each with value in the set 0, 1, …, m–1. That is, let k = ⟨k0, k1, …, km–1⟩, where 0 ≤ ki < m.Pick a = ⟨a0, a1, …, am–1⟩ where each ai is chosen randomly from 0, 1, …, m–1.
mkakhr
iiia mod)(
0∑=
=Define .
• Excellent in practice, but expensive to compute.
Introduction to Algorithms6.046J/18.401J
Lecture 8Prof. Piotr Indyk
Data structures
• Previous lecture: hash tables
– Insert, Delete, Search in (expected) constant time
– Works for integers from 0…mr-1 • This lecture: Binary Search Trees
– Insert, Delete, Search (Successor) – Works in comparison model
© Piotr Indyk Introduction to Algorithms October 6, 2004 L7.2
Binary Search Tree
• Each node x has:– key[x]
– Pointers: • left[x] • right[x] • p[x]
9
125
1 6
7
8
© Piotr Indyk Introduction to Algorithms October 6, 2004 L7.3
Binary Search Tree (BST)
• Property: for any node x:– For all nodes y in the left
subtree of x: key[y] ≤ key[x]
– For all nodes y in the rightsubtree of x:
key[y] ≥ key[x]
• Given a set of keys, is BST forthose keys unique?
9
125
1 6
7
8
© Piotr Indyk Introduction to Algorithms October 6, 2004 L7.4
No uniqueness
9
125
1 6
7
8
7
5 9
1 6 8 12
© Piotr Indyk Introduction to Algorithms October 6, 2004 L7.5
What can we do given BST ?
• Sort !• Inorder-Walk(x):
If x≠NIL then– Inorder-Walk( left[x] ) – print key[x] – Inorder-Walk( right[x] )
• Output: 1 765 8
9
125
1 6
7
89 12
© Piotr Indyk Introduction to Algorithms October 6, 2004 L7.6
Sorting, ctd.
• What is the running time of Inorder-Walk?
• It is O(n) • Because:
– Each link is traversed twice
– There are O(n) links
9
125
1 6
7
8
© Piotr Indyk Introduction to Algorithms October 6, 2004 L7.7
Sorting, ctd.
• Does it mean that we can sort n keys in O(n) time ?
• No
• It just means that building a BST takes Ω(n log n)time (in the comparison model)
9
125
1 6
7
8
© Piotr Indyk Introduction to Algorithms October 6, 2004 L7.8
BST as a data structure
• Operations: – Insert(x) – Delete(x) – Search(k)
© Piotr Indyk Introduction to Algorithms October 6, 2004 L7.9
Search
Search(x): • If x≠NIL then
– If key[x] = k then return x – If k < key[x] then return
Search( left[x] )– If k > key[x] then return
Search( right[x] )• Else return NIL
Search(8.5):
9
125
1 6
7
Search(8):
© Piotr Indyk Introduction to Algorithms October 6, 2004 L7.10
8
Predecessor/Successor
• Can modify Search (into Search’) such that,if k is not stored in BST, we get x such that: – Either it has the largest key[x]<k, or– It has the smallest key[x]>k
• Useful when k prone to errors • What if we always want a successor of k ?
– x=Search’(k) – If key[x]<k, then return Successor(x) – Else return x
© Piotr Indyk Introduction to Algorithms October 6, 2004 L7.11
Successor
Successor(x):• If right[x] ≠ NIL then
return Minimum( right[x] ) • Otherwise
– y ← p[x] – While y≠NIL and x=right[y] do
• x ← y • y ← p[y]
– Return y
9
125
1 6
7
8
y
y
x
y
x
y
x
9
© Piotr Indyk Introduction to Algorithms October 6, 2004 L7.12
x
Minimum
Minimum( x )• While left[x]≠NIL do
– x ← left[x] • Return x
9
125
1 6
7
8
© Piotr Indyk Introduction to Algorithms October 6, 2004 L7.13
Nearest Neighbor
• Assuming keys are numbers• For a key k, can we find x such that |k-key[x]| is
minimal ? • Yes:
– key[x] must be either a predecessor orsuccessor of k
– y=Search’(k) //y is either succ or pred of k – y’ =Successor(y) – y’’=Predecessor(y) – Report the closest of key[y], key[y’], key[y’’]
© Piotr Indyk Introduction to Algorithms October 6, 2004 L7.14
Analysis
• How much time does all of this take ?
• Worst case: O(height) • Height really important • Tree better be balanced
9
125
1 6
7
8
© Piotr Indyk Introduction to Algorithms October 6, 2004 L7.15
Constructing BST
Insert(z): • y ← NIL• x ← root• While x ≠ NIL do
– y ← x– If key[z] < key[x]
then x ← left[x]else x ← right[x]
• p[z] ← y• If key[z] < key[y]
then left[y] ← zelse right[y] ← z
October 6, 2004 L7.16
9
125
1 6
7
8 Insert(8.5) Insert(5.5)
8.5
5.5 y
z
© Piotr Indyk Introduction to Algorithms
Analysis
• After we insert n elements, what is the worst possible BST height ?
• Pretty bad: n-1
1 2
3 4
5 6
© Piotr Indyk Introduction to Algorithms October 6, 2004 L7.17
Average case analysis
• Consider keys 1,2,…,n, in a random order • Each permutation equally likely • For each key perform Insert • What is the likely height of the tree ? • It is O(log n)
© Piotr Indyk Introduction to Algorithms October 6, 2004 L7.18
Introduction to Algorithms October 6, 2004 L7.19© Piotr Indyk
Creating a random BST
1 2 3• n=9 33 4 5 6 7 8 9
1 2 4 5 6 7 8 96
4 5 7 8 98
7 9
5
4
1
22
4 7 9
3 6 8 5 1 2 7 4 9
Observations
• Each edge corresponds to a random partition
• Element x has height h ⇒ x participated inh partitions
• Let hx be a random variable denoting height of x
• What is Pr[hx >t] , where t=c lg n ?
© Piotr Indyk Introduction to Algorithms October 6, 2004 L7.20
Partitions
• A partition is lucky if the ratio is at least 1:3, i.e., each side has size ≥ 25%
• Probability of lucky partition is ½
• After log4/3 n lucky partitions the element becomes a leaf
• hx>t ⇒ in t= c log4/3 n partitions we had <log4/3 n lucky ones
• Toss t= c log4/3 n coins, what is the probability you get <k=log4/3 n heads ?
© Piotr Indyk Introduction to Algorithms October 6, 2004 L7.21
Concentration inequalities
• CLRS, p. 1118: probability of at most k heads t /2t-kin t trials is at most ( ) t k
Pr[hx >t] ≤ ( )/2t-kk
≤ (et/k)k/2t-k
= (ce)log4/3 n/2(c-1) log4/3 n
= 2lg(ce) log4/3 n/2 (c-1) log4/3 n
= 2 [lg(ce) – (c-1)] * (lg n)/ lg(4/3)
≤ 2 -1.1 lg n = 1/n1.1, for sufficient c
© Piotr Indyk Introduction to Algorithms October 6, 2004 L7.22
Final Analysis
• We know that for each x, Pr[hx >t] ≤ 1/n1.1
• We want Pr[h1>t or h2>t or … or hn>t] • This is at most
Pr[h1>t]+Pr[h2>t] +…+ Pr[hn>t] ≤ n * 1/n1.1
= 1/n0.1
• As n grows, probability of height >c lgnbecomes arbitrarily small
© Piotr Indyk Introduction to Algorithms October 6, 2004 L7.23
Summing up
• We have seen BSTs
• Support Search, Successor, Nearest Neighbor etc, as well as Insert
• Worst case: O(n) • But O(log n) on average • Next week: O(log n) worst case
© Piotr Indyk Introduction to Algorithms October 6, 2004 L7.24
Introduction to Algorithms6.046J/18.401J
Lecture 9Prof. Piotr Indyk
Today
• or how to avoid this Balanced search trees,
even in the worst case 1
2 3
4 5
6
© Piotr Indyk and Charles E. Leiserson Introduction to Algorithms October 13, 2004 L9.2
Balanced search treesBalanced search tree: A search-tree data structure for which a height of O(lg n) is guaranteed when implementing a dynamic set of n items.
• AVL trees
Examples: • 2-3 trees • 2-3-4 trees • B-trees • Red-black trees
© Piotr Indyk and Charles E. Leiserson Introduction to Algorithms October 13, 2004 L9.3
Red-black trees
BSTs with an extra one-bit color field in each node. Red-black properties: 1. Every node is either red or black. 2. The root and leaves (NIL’s) are black. 3. If a node is red, then its parent is black.4. All simple paths from any node x to a
descendant leaf have the same number of black nodes.
© Piotr Indyk and Charles E. Leiserson Introduction to Algorithms October 13, 2004 L9.4
Example of a red-black tree
88 1111
1010
1818
2626
2222
33
77
NIL NIL
NIL
NIL NIL NIL NIL NIL NIL
© Piotr Indyk and Charles E. Leiserson Introduction to Algorithms October 13, 2004 L9.5
Use of red-black trees
• What properties would we like to prove about red-black trees ? – They always have O(log n) height– There is an O(log n)–time insertion
procedure which preserves the red-black properties
• Is it true that, after we add a new element to a tree (as in the previous lecture), we can always recolor the tree to keep it red-black ?
© Piotr Indyk and Charles E. Leiserson Introduction to Algorithms October 13, 2004 L9.6
Example of a red-black tree
88 1111
1010
1818
2626
2222
33
77
7.57.5
© Piotr Indyk and Charles E. Leiserson Introduction to Algorithms October 13, 2004 L9.7
Use of red-black trees
• What properties would we like to prove about red-black trees ?– They always have O(log n) height– There is an O(log n)–time insertion procedure
which preserves the red-black properties • Is it true that, after we add a new element to a tree (as
in the previous lecture), we can always recolor thetree to keep it red-black ?
• NO • After insertions, sometimes we need to juggle nodes
around
© Piotr Indyk and Charles E. Leiserson Introduction to Algorithms October 13, 2004 L9.8
Rotations
AABB
αα ββ γγ
RIGHT-ROTATE(B)
BBAA
γγββ αα
LEFT-ROTATE(A)
Rotations maintain the inorder ordering of keys:• a ∈ α, b ∈ β, c ∈ γ ⇒ a ≤ A ≤ b ≤ B ≤ c.A rotation can be performed in O(1) time.
© Piotr Indyk and Charles E. Leiserson Introduction to Algorithms October 13, 2004 L9.9
Rotations can reduce height
B A
A
1 2
31
LEFT-ROTATE(A) B2 3 γ γ
AA
BB
αα ββ γγ
BB
AA
γγββ αα
© Piotr Indyk and Charles E. Leiserson Introduction to Algorithms October 13, 2004 L9.10
Red-black tree wrap-up
• Can show how
– O(log n) re-colorings
– 1 rotation
can restore red-black properties after an insertion
• Instead, we will see 2-3 trees (but will come back to red-black trees at the end)
© Piotr Indyk and Charles E. Leiserson Introduction to Algorithms October 13, 2004 L9.11
2-3 Trees
• The simplest balanced trees on the planet! • Although a little bit more wasteful
© Piotr Indyk and Charles E. Leiserson Introduction to Algorithms October 13, 2004 L9.12
2-3 Trees
• either 2 or 3
••
depth • Leaves are sorted
Degree of each node is
Keys are in the leaves All leaves have equal
9 1251 6 7 8
6 8 12
12
• Each node x contains maximum key in thesub-tree, denoted x.max
© Piotr Indyk and Charles E. Leiserson Introduction to Algorithms October 13, 2004 L9.13
Internal nodes
• Internal nodes:– Values:
• x.max: maximum key in the sub-tree – Pointers:
• left[x] • mid[x] • right[x] : can be null • p[x] : can be null for the root • …
• Leaves: – x.max : the key
© Piotr Indyk and Charles E. Leiserson Introduction to Algorithms October 13, 2004 L9.14
Height of 2-3 tree
• What is the maximum height h of a 2-3 tree with n nodes ?
• Alternatively, what is the minimum number of nodes in a 2-3 tree of height h ?
• It is 1+2+22+23+…+2h =2h+1-1 • n ≥ 2h+1-1 ⇒ h = O(log n) • Full binary tree is the worst-case example!
© Piotr Indyk and Charles E. Leiserson Introduction to Algorithms October 13, 2004 L9.15
Searching
• How can we search for a key k ?
Search(x,k): • If x=NIL then return NIL • Else if x is a leaf then
– If x.max=k then return x – Else return NIL
• Else 9 1251 6 7 8
6 8 12
12
– If k ≤ left[x].max then Search(left[x],k)
– Else if k ≤ mid[x].max Search(8)then Search(mid[x],k) Search(13)
– Else Search(right[x],k)
© Piotr Indyk and Charles E. Leiserson Introduction to Algorithms October 13, 2004 L9.16
12
12
Insertion
• How to insert x ? • Perform Search for the
key of x• Let y be the last internal
node • Insert x into y in a
sorted order• At the end, update the
max values on the pathto root
9 1251 6 7 8
6 8
7.55.5
13
13
Insert(7.5)(continued on the next Insert(13)slide) Insert(5.5)
© Piotr Indyk and Charles E. Leiserson Introduction to Algorithms October 13, 2004 L9.17
13
12
12
Insertion, ctd.
(continued from the previous slide)
• If y has 4 children, then Split(y)
x
9 1251 6 7 8
6 8
7.55.5
13
13
y
© Piotr Indyk and Charles E. Leiserson Introduction to Algorithms October 13, 2004 L9.18
13
Split
• Split y into two nodes y1, y2
• Both are linked to *z=parent(y)
• If z has 4 children, split z
*If y is a root, then create new parent(y)=new root
ba d
y
c
ba d
y1
c
y2
z
z
© Piotr Indyk and Charles E. Leiserson Introduction to Algorithms October 13, 2004 L9.19
12
12
Split
9 1251 6 7 8
6 8
5.5
13
13
7.5 13
© Piotr Indyk and Charles E. Leiserson Introduction to Algorithms October 13, 2004 L9.20
12
12
Split
9 1251 6 7 8
5.5 8
5.5
13
13
6
7.5 13
© Piotr Indyk and Charles E. Leiserson Introduction to Algorithms October 13, 2004 L9.21
12
Split
9 1251 6 7 8
5.5 8
5.5
13
6
6
13
13
7.5 13 • Insert and Split preserve heights, unless new root is created, in which case all heights are
increased by 1• After Split, all nodes have 2 or 3 children • Everything takes O(log n) time
© Piotr Indyk and Charles E. Leiserson Introduction to Algorithms October 13, 2004 L9.22
12
Delete
• How to delete x ? • Let y=p(x) • Remove x from y • If y has 1 child:
– Remove y 9 1251 6 7 8
5.5 8
5.5
12
6
6
12
12
y z
– Attach x to y’s sibling z x
Delete(8)
© Piotr Indyk and Charles E. Leiserson Introduction to Algorithms October 13, 2004 L9.23
12
Delete
• How to delete x ? • Let y=p(x) • Remove x from y • If y has 1 child:
– Remove y 9 1251 6 7
5.5
5.5
12
6
6
12
12
z
– Attach x to y’s sibling z • If z has 4 children, then
Split(z) Delete(8)INCOMPLETE – SEE THE END FOR FULL VERSION© Piotr Indyk and Charles E. Leiserson Introduction to Algorithms October 13, 2004 L9.24
Summing up
• 2-3 Trees: – O(log n) depth ⇒ Search in O(log n) – Insert, Delete (and Split) in O(log n)
• We will now see 2-3-4 trees – Same idea, but:
• Each parent has 2,3 or 4 children • Keys in the inner nodes • More complicated procedures
timetime
© Piotr Indyk and Charles E. Leiserson Introduction to Algorithms October 13, 2004 L9.25
2-3-4 Trees
5 9
1 2 4 7 8 10 12
© Piotr Indyk and Charles E. Leiserson Introduction to Algorithms October 13, 2004 L9.26
Height of a red-black tree
Theorem. A red-black tree with n keys has heighth ≤ 2 lg(n + 1).
INTUITION:• Merge red nodes
into their black parents.
© Piotr Indyk and Charles E. Leiserson Introduction to Algorithms October 13, 2004 L9.27
Height of a red-black tree
Theorem. A red-black tree with n keys has heighth ≤ 2 lg(n + 1).
INTUITION:• Merge red nodes
into their black parents.
© Piotr Indyk and Charles E. Leiserson Introduction to Algorithms October 13, 2004 L9.28
Height of a red-black tree
Theorem. A red-black tree with n keys has heighth ≤ 2 lg(n + 1).
INTUITION:• Merge red nodes
into their black parents.
© Piotr Indyk and Charles E. Leiserson Introduction to Algorithms October 13, 2004 L9.29
Height of a red-black tree
Theorem. A red-black tree with n keys has heighth ≤ 2 lg(n + 1).
INTUITION:• Merge red nodes
into their black parents.
© Piotr Indyk and Charles E. Leiserson Introduction to Algorithms October 13, 2004 L9.30
Height of a red-black tree
Theorem. A red-black tree with n keys has heighth ≤ 2 lg(n + 1).
INTUITION:• Merge red nodes
into their black parents.
© Piotr Indyk and Charles E. Leiserson Introduction to Algorithms October 13, 2004 L9.31
Height of a red-black tree
Theorem. A red-black tree with n keys has heighth ≤ 2 lg(n + 1).
INTUITION:• Merge red nodes
into their black parents.
h′
• This process produces a tree in which each node has 2, 3, or 4 children.
• The 2-3-4 tree has uniform depth h′ of leaves. © Piotr Indyk and Charles E. Leiserson Introduction to Algorithms October 13, 2004 L9.32
Summing up
• We have seen: – Red-black trees – 2-3 trees (in detail) – 2-3-4 trees
• Red-black trees are undercover 2-3-4 trees • In most cases, does not matter what you use
© Piotr Indyk and Charles E. Leiserson Introduction to Algorithms October 13, 2004 L9.33
12
2-3 Trees: Deletions
• Problem: there is an internal node that has only 1 child
9 1251 6 7
5.5
5.5
12
6
6
12
12
© Piotr Indyk and Charles E. Leiserson Introduction to Algorithms October 13, 2004 L9.34
Full procedure for Delete(x)
• Special case: x is the only element in the tree: delete everything
x NIL• Not-so-special case: x is one of two elements
in the tree. In this case, the procedure on thenext slide will delete x
x y
y
• Both NIL and y are special 2-3 trees © Piotr Indyk and Charles E. Leiserson Introduction to Algorithms October 13, 2004 L9.35
Procedure for Delete(x)• Let y=p(x) • Remove x • If y≠root then
– Let z be the sibling of y. – Assume z is the right sibling of y, otherwise the code is
symmetric.– If y has only 1 child w left
Case 1: z has 3 children • Attach left[z] as the rightmost child of y • Update y.max and z.max Case 2: z has 2 children: • Attach the child w of y as the leftmost child of z • Update z.max • Delete(y) (recursively*)
– Else • Update max of y, p(y), p(p(y)) and so on until root
• Else – If root has only one child u
• Remove root • Make u the new root
*Note that the input of Delete does not have to be a leaf
x
zy
© Piotr Indyk and Charles E. Leiserson Introduction to Algorithms October 13, 2004 L9.36
12
Example
9 1251 6 7 8
5.5 8
5.5
12
6
6
12
12
© Piotr Indyk and Charles E. Leiserson Introduction to Algorithms October 13, 2004 L9.37
12
Example, ctd.
9 1251 6 7
5.5 8
5.5
12
6
6
12
12
© Piotr Indyk and Charles E. Leiserson Introduction to Algorithms October 13, 2004 L9.38
12
Example, ctd.
9 1251 6 7
5.5
5.5
12
12
6
12
12
© Piotr Indyk and Charles E. Leiserson Introduction to Algorithms October 13, 2004 L9.39
12
Example, ctd.
9 1251 6 7
5.5
5.5
12
12
6
© Piotr Indyk and Charles E. Leiserson Introduction to Algorithms October 13, 2004 L9.40
Introduction to Algorithms6.046J/18.401J/SMA5503
Lecture 10Prof. Piotr Indyk
Introduction to Algorithms October 18, 2004 L10.2© 2004 by Erik Demaine and Piotr Indyk
Today
• A data structure for a new problem• Amortized analysis
Introduction to Algorithms October 18, 2004 L10.3© 2004 by Erik Demaine and Piotr Indyk
2-3 Trees: Deletions
• Problem: there is an internal node that has only 1child
• Solution: delete recursively
9 1251 6 7
5.5 12
5.5
12
6
6
12
12
Introduction to Algorithms October 18, 2004 L10.4© 2004 by Erik Demaine and Piotr Indyk
Example
9 1251 6 7 8
5.5 8 12
5.5
12
6
6
12
12
Introduction to Algorithms October 18, 2004 L10.5© 2004 by Erik Demaine and Piotr Indyk
Example, ctd.
9 1251 6 7
5.5 8 12
5.5
12
6
6
12
12
Introduction to Algorithms October 18, 2004 L10.6© 2004 by Erik Demaine and Piotr Indyk
Example, ctd.
9 1251 6 7
5.5 12
5.5
12
12
6
12
12
Introduction to Algorithms October 18, 2004 L10.7© 2004 by Erik Demaine and Piotr Indyk
Example, ctd.
9 1251 6 7
5.5 12
5.5
12
12
6
Introduction to Algorithms October 18, 2004 L10.8© 2004 by Erik Demaine and Piotr Indyk
Procedure for Delete(x)• Let y=p(x)• Remove x• If y≠root then
– Let z be the sibling of y.– Assume z is the right sibling of y, otherwise the code is
symmetric.– If y has only 1 child w left
Case 1: z has 3 children • Attach left[z] as the rightmost child of y• Update y.max and z.maxCase 2: z has 2 children:• Attach the child w of y as the leftmost child of z• Update z.max• Delete(y) (recursively*)
– Else• Update max of y, p(y), p(p(y)) and so on until root
• Else – If root has only one child u
• Remove root• Make u the new root
*Note that the input of Delete does not have to be a leaf
x
zy
Introduction to Algorithms October 18, 2004 L10.9© 2004 by Erik Demaine and Piotr Indyk
2-3 Trees
• The simplest balanced trees on the planet!(but, nevertheless, not completely trivial)
Introduction to Algorithms October 18, 2004 L10.10© 2004 by Erik Demaine and Piotr Indyk
Dynamic Maintenance of Sets• Assume, we have a collection
of elements• The elements are clustered• Initially, each element forms
its own cluster/set• We want to enable two
operations:– FIND-SET(x): report the
cluster containing x– UNION(C1, C2): merges
the clusters C1, C2
1 2
3
5
6
Introduction to Algorithms October 18, 2004 L10.11© 2004 by Erik Demaine and Piotr Indyk
Disjoint-set data structure(Union-Find)
Problem:• Maintain a collection of pairwise-disjointsets S = S1, S2, …, Sr.• Each Si has one representative element x=rep[Si].• Must support three operations:
• MAKE-SET(x): adds new set x to Swith rep[x] = x (for any x ∉ Si for all i).
• UNION(x, y): replaces sets Sx, Sy with Sx ∪ Syin S for any x, y in distinct sets Sx, Sy .
• FIND-SET(x): returns representative rep[Sx]of set Sx containing element x.
rep.WEAK
Introduction to Algorithms October 18, 2004 L10.12© 2004 by Erik Demaine and Piotr Indyk
Quiz
• If we have a WEAKUNION( x, y) that works only if x, y are representatives, how can we implement UNION that works for any x, y ?
• UNION( x, y)=WEAKUNION( FIND-SET(x) , FIND-SET(y) )
Introduction to Algorithms October 18, 2004 L10.13© 2004 by Erik Demaine and Piotr Indyk
Representation
x
Other fields containingdata of our choice
Data
Introduction to Algorithms October 18, 2004 L10.14© 2004 by Erik Demaine and Piotr Indyk
Applications
• Data clustering• Killer App: Minimum
Spanning Tree (Lecture 13)
• Amortized analysis
1 2
3
5
6
Introduction to Algorithms October 18, 2004 L10.15© 2004 by Erik Demaine and Piotr Indyk
Ideas ?
• How can we implement this data structure efficiently ?– MAKE-SET
– UNION
– FIND-SET
Introduction to Algorithms October 18, 2004 L10.16© 2004 by Erik Demaine and Piotr Indyk
Bad case for UNION or FIND
1 2 … n n+1 … 2n
Introduction to Algorithms October 18, 2004 L10.17© 2004 by Erik Demaine and Piotr Indyk
Simple linked-list solutionStore set Si = x1, x2, …, xk as an (unordered) doubly linked list. Define representative elementrep[Si] to be the front of the list, x1.
…Si : x1 x2 xk
rep[Si]
• MAKE-SET(x) initializes x as a lone node.• FIND-SET(x) walks left in the list containing x
until it reaches the front of the list.• UNION(x, y) concatenates the lists containing
x and y, leaving rep. as FIND-SET[x].
Θ(1)
Θ(n)
Θ(n)
How can we improve it ?
Introduction to Algorithms October 18, 2004 L10.18© 2004 by Erik Demaine and Piotr Indyk
Augmented linked-list solution
…Si : x1 x2 xk
rep[Si]
Store set Si = x1, x2, …, xk as unordered doublylinked list. Each xj also stores pointer rep[xj] to head.
• FIND-SET(x) returns rep[x].• UNION(x, y) concatenates the lists containing
x and y, and updates the rep pointers forall elements in the list containing y.
Introduction to Algorithms October 18, 2004 L10.19© 2004 by Erik Demaine and Piotr Indyk
Example ofaugmented linked-list solution
Sx : x1 x2
rep[Sx]
rep
Sy : y1 y2 y3
rep[Sy]
rep
Introduction to Algorithms October 18, 2004 L10.20© 2004 by Erik Demaine and Piotr Indyk
Example ofaugmented linked-list solution
Sx ∪ Sy :x1 x2
rep[Sx]
rep
y1 y2 y3
rep[Sy]
rep
Introduction to Algorithms October 18, 2004 L10.21© 2004 by Erik Demaine and Piotr Indyk
Example ofaugmented linked-list solution
Sx ∪ Sy :
x1 x2
rep[Sx ∪ Sy]y1 y2 y3
rep
Introduction to Algorithms October 18, 2004 L10.22© 2004 by Erik Demaine and Piotr Indyk
Augmented linked-list solution
…Si : x1 x2 xk
rep[Si]
Store set Si = x1, x2, …, xk as unordered doublylinked list. Each xj also stores pointer rep[xj] to head.
• FIND-SET(x) returns rep[x].• UNION(x, y) concatenates the lists containing
x and y, and updates the rep pointers forall elements in the list containing y.
Θ(1)
Θ(n)
?
Introduction to Algorithms October 18, 2004 L10.23© 2004 by Erik Demaine and Piotr Indyk
Amortized analysis
• So far, we focused on worst-case time of each operation.– E.g., UNION takes Θ(n) time for some operations
• Amortized analysis: count the total time spent by any sequence of operations
• Total time is always at mostworst-case-time-per-operation * #operations
but it can be much better!• E.g., if times are 1,1,1,…,1,n,1,…,1• Can we modify the linked-list data structure so that any
sequence of m MAKE-SET, FIND-SET, UNION operations cost less than m*Θ(n) time?
Introduction to Algorithms October 18, 2004 L10.24© 2004 by Erik Demaine and Piotr Indyk
Alternative
Sx : x1 x2
rep[Sy]
UNION(x, y) :• concatenates the lists containing y and x, and• update the rep pointers for all elements in the
list containing
y1 y2 y3
rep
rep[Sx]rep
Sy :
xy
Introduction to Algorithms October 18, 2004 L10.25© 2004 by Erik Demaine and Piotr Indyk
Alternative concatenation
Sx ∪ Sy :x1 x2
rep[Sy]
UNION(x, y) could instead• concatenate the lists containing y and x, and• update the rep pointers for all elements in the
list containing x.
y1 y2 y3
rep[Sx]rep
rep
Introduction to Algorithms October 18, 2004 L10.26© 2004 by Erik Demaine and Piotr Indyk
Alternative concatenation
Sx ∪ Sy :x1 x2
UNION(x, y) could instead• concatenate the lists containing y and x, and• update the rep pointers for all elements in the
list containing x.
y1 y2 y3
rep
rep
rep[Sx ∪ Sy]
Introduction to Algorithms October 18, 2004 L10.27© 2004 by Erik Demaine and Piotr Indyk
Smaller into larger• Concatenate smaller list onto the end of the larger list (each list stores its weight = # elements)
• Cost = Θ(length of smaller list).
Let n denote the overall number of elements(equivalently, the number of MAKE-SET operations).Let m denote the total number of operations.
Theorem: Cost of all UNION’s is O(n lg n).Corollary: Total cost is O(m + n lg n).
Introduction to Algorithms October 18, 2004 L10.28© 2004 by Erik Demaine and Piotr Indyk
Total UNION cost is O(n lg n)Proof:• Monitor an element x and set Sx containing it• After initial MAKE-SET(x), weight[Sx] = 1• Consider any time when Sx is merged with set Sy
– If weight[Sy] ≥ weight[Sx]• pay 1 to update rep[x]• weight[Sx] at least doubles (increasing by weight[Sy])
– Otherwise• pay nothing• weight[Sx] only increases
• Thus:– Each time we pay 1, the weight doubles– Maximum possible weight is n– Maximum pay ≤ lg n for x , or O(n log n) overall
Introduction to Algorithms October 18, 2004 L10.29© 2004 by Erik Demaine and Piotr Indyk
Final Result
• We have a data structure for dynamic sets which supports:– MAKE-SET: O(1) worst case– FIND-SET: O(1) worst case– UNION:
• Any sequence of any m operations* takes O(m log n) time, or• … the amortized complexity of the operations* is O(log n)
* I.e., MAKE-SET, FIND-SET or UNION
Introduction to Algorithms October 18, 2004 L10.30© 2004 by Erik Demaine and Piotr Indyk
Amortized vs Average
• What is the difference between average case complexity and amortized complexity ?– “Average case” assumes random
distribution over the input (e.g., random sequence of operations)
– “Amortized” means we count the totaltime taken by any sequence of moperations (and divide it by m)
Introduction to Algorithms October 18, 2004 L10.31© 2004 by Erik Demaine and Piotr Indyk
Can we do better ?
• One can do:– MAKE-SET: O(1) worst case– FIND-SET: O(lg n) worst case– WEAKUNION: O(1) worst case– Thus, UNION: O(lg n) worst case
Introduction to Algorithms October 18, 2004 L10.32© 2004 by Erik Demaine and Piotr Indyk
Representing sets as trees• Each set Si = x1, x2, …, xk stored as a tree• rep[Si] is the tree root.
S1 = x1, x2, x3, x4, x5 , x6S2 = x7
• MAKE-SET(x) initializes xas a lone node.
• FIND-SET(x) walks up thetree containing x until itreaches the root.
• UNION(x, y) concatenatesthe trees containingx and y
x1
x4 x3
x2 x5
rep[S1]
x6
x7
rep[S2]
UNION(rep[S1] ,rep[S1]): rep[S1 ∪ S2]
Introduction to Algorithms October 18, 2004 L10.33© 2004 by Erik Demaine and Piotr Indyk
Time Analysis
O(1)• MAKE-SET(x) initializes xas a lone node.
• FIND-SET(x) walks up thetree containing x until itreaches the root.
• WEAKUNION(x, y)concatenatesthe trees containing x and y
O(depth)
O(1)
= ?
Introduction to Algorithms October 18, 2004 L10.34© 2004 by Erik Demaine and Piotr Indyk
“Smaller into Larger” in trees
y1
y4 y3
Algorithm: Merge tree with smaller weight into tree withlarger weight.• Height of tree increases only when its sizedoubles• Height logarithmic in weight
x1
x4 x3
x2 x5 x6
Introduction to Algorithms October 18, 2004 L10.35© 2004 by Erik Demaine and Piotr Indyk
“Smaller into Larger” in trees
Proof:• Monitor the height of an element z• Each time the height of z increases, the
weight of its tree doubles• Maximum weight is n• Thus, height of z is ≤log n
Introduction to Algorithms October 18, 2004 L10.36© 2004 by Erik Demaine and Piotr Indyk
Tree implementation
• We have:– MAKE-SET: O(1) worst case– FIND-SET: O(depth) = O(lg n) worst case– WEAKUNION: O(1) worst case
• Can amortized analysis buy us anything ? • Need another trick…
Introduction to Algorithms October 18, 2004 L10.37© 2004 by Erik Demaine and Piotr Indyk
Trick 2: Path compressionWhen we execute a FIND-SET operation and walkup a path to the root, we know the representativefor all the nodes on the path.
y1
y4 y3
y2 y5
x1
x4 x3
x2 x5 x6
Path compression makesall of those nodes directchildren of the root.
FIND-SET(y2)
Introduction to Algorithms October 18, 2004 L10.38© 2004 by Erik Demaine and Piotr Indyk
Trick 2: Path compressionWhen we execute a FIND-SET operation and walkup a path to the root, we know the representativefor all the nodes on the path.
y1
y4 y3
y2 y5
x1
x4 x3
x2 x5 x6
Path compression makesall of those nodes directchildren of the root.
FIND-SET(y2)
Introduction to Algorithms October 18, 2004 L10.39© 2004 by Erik Demaine and Piotr Indyk
Trick 2: Path compressionWhen we execute a FIND-SET operation and walkup a path p to the root, we know the representativefor all the nodes on path p.
y1
y4
y3y2
y5
x1
x4 x3
x2 x5 x6
FIND-SET(y2)
Path compression makesall of those nodes directchildren of the root.
Cost of FIND-SET(x)is still Θ(depth[x]).
Introduction to Algorithms October 18, 2004 L10.40© 2004 by Erik Demaine and Piotr Indyk
Theorem: In general, amortized cost is
The Theorem
where α(n) grows really, really, really slow.O(α(n)),
Introduction to Algorithms October 18, 2004 L10.41© 2004 by Erik Demaine and Piotr Indyk
Ackermann’s function A
Define
≥=+
= +− 1 if
0 if )(
1)( )1(
1 kk
jAj
jA jk
k
Define α(n) = min k : Ak(1) ≥ n.
A0( j) = j + 1A1( j) = A0(…(A0( j)…) ~2jA2( j) = A1(…A1( j)…) ~2j 2j
A3( j ) >
A4( j) is a lot bigger.
22
2
2 j
...
j
A0(1) = 2A1(1) = 3A2(1) = 7A3(1) = 2047
A4(1) >
-iterate Ak-1() j+1 times
22
2
22047
...
2048
Introduction to Algorithms October 18, 2004 L10.42© 2004 by Erik Demaine and Piotr Indyk
Theorem: In general, amortized cost is
The Theorem
where α(n) grows really, really, really slow.O(α(n)),
Proof: Really, really, really long (CLRS, p. 509)
Introduction to Algorithms October 18, 2004 L10.43© 2004 by Erik Demaine and Piotr Indyk
Application:Dynamic connectivity
Suppose a graph is given to us incrementally by• ADD-VERTEX(v)• ADD-EDGE(u, v)
and we want to support connectivity queries:• CONNECTED(u, v):Are u and v in the same connected component?
For example, we want to maintain a spanning forest,so we check whether each new edge connects apreviously disconnected pair of vertices.
Introduction to Algorithms October 18, 2004 L10.44© 2004 by Erik Demaine and Piotr Indyk
Application:Dynamic connectivity
Sets of vertices represent connected components.Suppose a graph is given to us incrementally by
• ADD-VERTEX(v) – MAKE-SET(v)• ADD-EDGE(u, v) – if not CONNECTED(u, v)
then UNION(v, w)and we want to support connectivity queries:
• CONNECTED(u, v): – FIND-SET(u) = FIND-SET(v)Are u and v in the same connected component?
For example, we want to maintain a spanning forest,so we check whether each new edge connects apreviously disconnected pair of vertices.
Introduction to Algorithms October 18, 2004 L10.45© 2004 by Erik Demaine and Piotr Indyk
Simple balanced-tree solutionStore each set Si = x1, x2, …, xk as a balanced tree(ignoring keys). Define representative elementrep[Si] to be the root of the tree.
x1
x4 x3
x2 x5
• MAKE-SET(x) initializes xas a lone node.
• FIND-SET(x) walks up thetree containing x until itreaches the root.
• UNION(x, y) concatenatesthe trees containing x and y,changing rep.
Si = x1, x2, x3, x4, x5
rep[Si]– Θ(1)
– Θ(lg n)
– Θ(lg n)
Introduction to Algorithms October 18, 2004 L10.46© 2004 by Erik Demaine and Piotr Indyk
Plan of attackWe will build a simple disjoint-union data structurethat, in an amortized sense, performs significantlybetter than Θ(lg n) per op., even better thanΘ(lg lg n), Θ(lg lg lg n), etc., but not quite Θ(1).
To reach this goal, we will introduce two key tricks.Each trick converts a trivial Θ(n) solution into asimple Θ(lg n) amortized solution. Together, thetwo tricks yield a much better solution.
First trick arises in an augmented linked list.Second trick arises in a tree structure.
Introduction to Algorithms October 18, 2004 L10.47© 2004 by Erik Demaine and Piotr Indyk
Each element xj stores pointer rep[xj] to rep[Si].
UNION(x, y)• concatenates the lists containing x and y,
and• updates the rep pointers for all elements
in thelist containing y.
Introduction to Algorithms October 18, 2004 L10.48© 2004 by Erik Demaine and Piotr Indyk
Analysis of Trick 2 aloneTheorem: Total cost of FIND-SET’s is O(m lg n).Proof: Amortization by potential function.The weight of a node x is # nodes in its subtree.Define φ(x1, …, xn) = Σi lg weight[xi].UNION(xi, xj) increases potential of root FIND-SET(xi)by at most lg weight[root FIND-SET(xj)] ≤ lg n.Each step down p → c made by FIND-SET(xi),except the first, moves c’s subtree out of p’s subtree.Thus if weight[c] ≥ ½ weight[p], φ decreases by ≥ 1,paying for the step down. There can be at most lg nsteps p → c for which weight[c] < ½ weight[p].
Introduction to Algorithms October 18, 2004 L10.49© 2004 by Erik Demaine and Piotr Indyk
Analysis of Trick 2 aloneTheorem: If all UNION operations occur beforeall FIND-SET operations, then total cost is O(m).
Proof: If a FIND-SET operation traverses a pathwith k nodes, costing O(k) time, then k – 2 nodesare made new children of the root. This changecan happen only once for each of the n elements,so the total cost of FIND-SET is O(f + n).
Introduction to Algorithms October 18, 2004 L10.50© 2004 by Erik Demaine and Piotr Indyk
UNION(x, y)
• Every tree has a rank• Rank is an upper bound for height• When we take UNION(x, y):
– If rank[x] >rank[y] then link y to x– If rank[x] <rank[y] then link x to y– If rank[x]=rank[y] then
• link x to y• rank[y]=rank[y]+1
• Can show that 2rank(x) ≤ #elements in x (Exercise 21.4-2)• Therefore, height is O(log n)
Introduction to Algorithms6.046J/18.401J
Prof. Charles E. Leiserson
LECTURE 11Amortized analysis• Dynamic tables• Aggregate method• Accounting method• Potential method
Introduction to Algorithms October 20, 2004 L14.2© 2001–4 by Charles E. Leiserson
How large should a hash table be?
Problem: What if we don’t know the proper size in advance?
Goal: Make the table as small as possible, but large enough so that it won’t overflow (or otherwise become inefficient).
IDEA: Whenever the table overflows, “grow” it by allocating (via malloc or new) a new, larger table. Move all items from the old table into the new one, and free the storage for the old table.
Solution: Dynamic tables.
Introduction to Algorithms October 20, 2004 L14.3© 2001–4 by Charles E. Leiserson
Example of a dynamic table
1. INSERT 1
2. INSERT overflow
Introduction to Algorithms October 20, 2004 L14.4© 2001–4 by Charles E. Leiserson
11
Example of a dynamic table
1. INSERT2. INSERT overflow
Introduction to Algorithms October 20, 2004 L14.5© 2001–4 by Charles E. Leiserson
112
Example of a dynamic table
1. INSERT2. INSERT
Introduction to Algorithms October 20, 2004 L14.6© 2001–4 by Charles E. Leiserson
Example of a dynamic table
1. INSERT2. INSERT
11
22
3. INSERT overflow
Introduction to Algorithms October 20, 2004 L14.7© 2001–4 by Charles E. Leiserson
Example of a dynamic table
1. INSERT2. INSERT3. INSERT
21
overflow
Introduction to Algorithms October 20, 2004 L14.8© 2001–4 by Charles E. Leiserson
Example of a dynamic table
1. INSERT2. INSERT3. INSERT
21
Introduction to Algorithms October 20, 2004 L14.9© 2001–4 by Charles E. Leiserson
Example of a dynamic table
1. INSERT2. INSERT3. INSERT4. INSERT 4
321
Introduction to Algorithms October 20, 2004 L14.10© 2001–4 by Charles E. Leiserson
Example of a dynamic table
1. INSERT2. INSERT3. INSERT4. INSERT5. INSERT
4321
overflow
Introduction to Algorithms October 20, 2004 L14.11© 2001–4 by Charles E. Leiserson
Example of a dynamic table
1. INSERT2. INSERT3. INSERT4. INSERT5. INSERT
4321
overflow
Introduction to Algorithms October 20, 2004 L14.12© 2001–4 by Charles E. Leiserson
Example of a dynamic table
1. INSERT2. INSERT3. INSERT4. INSERT5. INSERT
4321
Introduction to Algorithms October 20, 2004 L14.13© 2001–4 by Charles E. Leiserson
Example of a dynamic table
1. INSERT2. INSERT3. INSERT4. INSERT
6. INSERT 65. INSERT 5
4321
77. INSERT
Introduction to Algorithms October 20, 2004 L14.14© 2001–4 by Charles E. Leiserson
Worst-case analysis
Consider a sequence of n insertions. The worst-case time to execute one insertion is Θ(n). Therefore, the worst-case time for ninsertions is n ·Θ(n) = Θ(n2).
WRONG! In fact, the worst-case cost for n insertions is only Θ(n) Θ(n2).
Let’s see why.
Introduction to Algorithms October 20, 2004 L14.15© 2001–4 by Charles E. Leiserson
Tighter analysis
i 1 2 3 4 5 6 7 8 9 10sizei 1 2 4 4 8 8 8 8 16 16
ci 1 2 3 1 5 1 1 1 9 1
Let ci = the cost of the i th insertion
= i if i – 1 is an exact power of 2,1 otherwise.
Introduction to Algorithms October 20, 2004 L14.16© 2001–4 by Charles E. Leiserson
Tighter analysis
Let ci = the cost of the i th insertion
= i if i – 1 is an exact power of 2,1 otherwise.
i 1 2 3 4 5 6 7 8 9 10sizei 1 2 4 4 8 8 8 8 16 16
1 1 1 1 1 1 1 1 1 11 2 4 8
ci
Introduction to Algorithms October 20, 2004 L14.17© 2001–4 by Charles E. Leiserson
Tighter analysis (continued)
)(3
2
)1lg(
0
1
nn
n
c
n
j
j
n
ii
Θ=≤
+≤
=
∑
∑−
=
=Cost of n insertions
.
Thus, the average cost of each dynamic-table operation is Θ(n)/n = Θ(1).
Introduction to Algorithms October 20, 2004 L14.18© 2001–4 by Charles E. Leiserson
Amortized analysisAn amortized analysis is any strategy for analyzing a sequence of operations to show that the average cost per operation is small, even though a single operation within the sequence might be expensive.
Even though we’re taking averages, however, probability is not involved!• An amortized analysis guarantees the
average performance of each operation in the worst case.
Introduction to Algorithms October 20, 2004 L14.19© 2001–4 by Charles E. Leiserson
Types of amortized analysesThree common amortization arguments:• the aggregate method,• the accounting method,• the potential method.We’ve just seen an aggregate analysis. The aggregate method, though simple, lacks the precision of the other two methods. In particular, the accounting and potential methods allow a specific amortized cost to be allocated to each operation.
Introduction to Algorithms October 20, 2004 L14.20© 2001–4 by Charles E. Leiserson
Accounting method• Charge i th operation a fictitious amortized costĉi, where $1 pays for 1 unit of work (i.e., time).
• This fee is consumed to perform the operation.• Any amount not immediately consumed is stored
in the bank for use by subsequent operations.• The bank balance must not go negative! We
must ensure that
∑∑==
≤n
ii
n
ii cc
11ˆ
for all n.• Thus, the total amortized costs provide an upper
bound on the total true costs.
Introduction to Algorithms October 20, 2004 L14.21© 2001–4 by Charles E. Leiserson
$0$0 $0$0 $0$0 $0$0 $2$2 $2$2
Example:$2 $2
Accounting analysis of dynamic tables
Charge an amortized cost of ĉi = $3 for the i th insertion.• $1 pays for the immediate insertion.• $2 is stored for later table doubling.When the table doubles, $1 pays to move a recent item, and $1 pays to move an old item.
overflow
Introduction to Algorithms October 20, 2004 L14.22© 2001–4 by Charles E. Leiserson
Example:
Accounting analysis of dynamic tables
Charge an amortized cost of ĉi = $3 for the i th insertion.• $1 pays for the immediate insertion.• $2 is stored for later table doubling.When the table doubles, $1 pays to move a recent item, and $1 pays to move an old item.
overflow
$0$0 $0$0 $0$0 $0$0 $0$0 $0$0 $0$0 $0$0
Introduction to Algorithms October 20, 2004 L14.23© 2001–4 by Charles E. Leiserson
Example:
Accounting analysis of dynamic tables
Charge an amortized cost of ĉi = $3 for the i th insertion.• $1 pays for the immediate insertion.• $2 is stored for later table doubling.When the table doubles, $1 pays to move a recent item, and $1 pays to move an old item.
$0$0 $0$0 $0$0 $0$0 $0$0 $0$0 $0$0 $0$0 $2 $2 $2
Introduction to Algorithms October 20, 2004 L14.24© 2001–4 by Charles E. Leiserson
Accounting analysis (continued)
Key invariant: Bank balance never drops below 0. Thus, the sum of the amortized costs provides an upper bound on the sum of the true costs.
i 1 2 3 4 5 6 7 8 9 10sizei 1 2 4 4 8 8 8 8 16 16
ci 1 2 3 1 5 1 1 1 9 1ĉi 2 3 3 3 3 3 3 3 3 3
banki 1 2 2 4 2 4 6 8 2 4*
*Okay, so I lied. The first operation costs only $2, not $3.
Introduction to Algorithms October 20, 2004 L14.25© 2001–4 by Charles E. Leiserson
Potential methodIDEA: View the bank account as the potential energy (à la physics) of the dynamic set.Framework:• Start with an initial data structure D0.• Operation i transforms Di–1 to Di. • The cost of operation i is ci.• Define a potential function Φ : Di → R,
such that Φ(D0 ) = 0 and Φ(Di ) ≥ 0 for all i. • The amortized cost ĉi with respect to Φ is
defined to be ĉi = ci + Φ(Di) – Φ(Di–1).
Introduction to Algorithms October 20, 2004 L14.26© 2001–4 by Charles E. Leiserson
Understanding potentialsĉi = ci + Φ(Di) – Φ(Di–1)
potential difference ∆Φi
• If ∆Φi > 0, then ĉi > ci. Operation i stores work in the data structure for later use.
• If ∆Φi < 0, then ĉi < ci. The data structure delivers up stored work to help pay for operation i.
Introduction to Algorithms October 20, 2004 L14.27© 2001–4 by Charles E. Leiserson
The amortized costs bound the true costs
The total amortized cost of n operations is
( )∑∑=
−=
Φ−Φ+=n
iiii
n
ii DDcc
11
1)()(ˆ
Summing both sides.
Introduction to Algorithms October 20, 2004 L14.28© 2001–4 by Charles E. Leiserson
The amortized costs bound the true costs
The total amortized cost of n operations is
( )
)()(
)()(ˆ
01
11
1
DDc
DDcc
n
n
ii
n
iiii
n
ii
Φ−Φ+=
Φ−Φ+=
∑
∑∑
=
=−
=
The series telescopes.
Introduction to Algorithms October 20, 2004 L14.29© 2001–4 by Charles E. Leiserson
The amortized costs bound the true costs
The total amortized cost of n operations is
( )
∑
∑
∑∑
=
=
=−
=
≥
Φ−Φ+=
Φ−Φ+=
n
ii
n
n
ii
n
iiii
n
ii
c
DDc
DDcc
1
01
11
1
)()(
)()(ˆ
since Φ(Dn) ≥ 0 andΦ(D0 ) = 0.
Introduction to Algorithms October 20, 2004 L14.30© 2001–4 by Charles E. Leiserson
Potential analysis of table doubling
Define the potential of the table after the ith insertion by Φ(Di) = 2i – 2lg i. (Assume that 2lg 0 = 0.)Note:• Φ(D0 ) = 0,• Φ(Di) ≥ 0 for all i.Example:
•• •• •• •• •• •• Φ = 2·6 – 23 = 4
$0$0 $0$0 $0$0 $0$0 $2$2 $2$2 accounting method)(
Introduction to Algorithms October 20, 2004 L14.31© 2001–4 by Charles E. Leiserson
Calculation of amortized costs
The amortized cost of the i th insertion is
ĉi = ci + Φ(Di) – Φ(Di–1)
i + (2i – 2lg i) – (2(i –1) – 2lg (i–1))if i – 1 is an exact power of 2,
1 + (2i – 2lg i) – (2(i –1) – 2lg (i–1))otherwise.
=
Introduction to Algorithms October 20, 2004 L14.32© 2001–4 by Charles E. Leiserson
Calculation (Case 1)
Case 1: i – 1 is an exact power of 2.
ĉi = i + (2i – 2lg i) – (2(i –1) – 2lg (i–1))= i + 2 – (2lg i – 2lg (i–1))= i + 2 – (2(i – 1) – (i – 1))= i + 2 – 2i + 2 + i – 1= 3
Introduction to Algorithms October 20, 2004 L14.33© 2001–4 by Charles E. Leiserson
Calculation (Case 2)
Case 2: i – 1 is not an exact power of 2.
ĉi = 1 + (2i – 2lg i) – (2(i –1) – 2lg (i–1))= 1 + 2 – (2lg i – 2lg (i–1))= 3
Therefore, n insertions cost Θ(n) in the worst case.
Exercise: Fix the bug in this analysis to show that the amortized cost of the first insertion is only 2.
Introduction to Algorithms October 20, 2004 L14.34© 2001–4 by Charles E. Leiserson
Conclusions• Amortized costs can provide a clean abstraction
of data-structure performance.• Any of the analysis methods can be used when
an amortized analysis is called for, but each method has some situations where it is arguably the simplest.
• Different schemes may work for assigning amortized costs in the accounting method, or potentials in the potential method, sometimes yielding radically different bounds.
Introduction to Algorithms6.046J/18.401J
Prof. Charles E. Leiserson
LECTURE 12Dynamic programming• Longest common
subsequence• Optimal substructure• Overlapping subproblems
Introduction to Algorithms October 25, 2004 L12.2© 2001–4 by Charles E. Leiserson
Dynamic programmingDesign technique, like divide-and-conquer.
Example: Longest Common Subsequence (LCS)• Given two sequences x[1 . . m] and y[1 . . n], find
a longest subsequence common to them both.
Introduction to Algorithms October 25, 2004 L12.3© 2001–4 by Charles E. Leiserson
Dynamic programmingDesign technique, like divide-and-conquer.
Example: Longest Common Subsequence (LCS)• Given two sequences x[1 . . m] and y[1 . . n], find
a longest subsequence common to them both.“a” not “the”
Introduction to Algorithms October 25, 2004 L12.4© 2001–4 by Charles E. Leiserson
Dynamic programmingDesign technique, like divide-and-conquer.
Example: Longest Common Subsequence (LCS)• Given two sequences x[1 . . m] and y[1 . . n], find
a longest subsequence common to them both.
x: A B C B D A B
y: B D C A B A
“a” not “the”
Introduction to Algorithms October 25, 2004 L12.5© 2001–4 by Charles E. Leiserson
Dynamic programmingDesign technique, like divide-and-conquer.
Example: Longest Common Subsequence (LCS)• Given two sequences x[1 . . m] and y[1 . . n], find
a longest subsequence common to them both.
x: A B C B D A B
y: B D C A B A
“a” not “the”
BCBA = LCS(x, y)
functional notation, but not a function
Introduction to Algorithms October 25, 2004 L12.6© 2001–4 by Charles E. Leiserson
Brute-force LCS algorithm
Check every subsequence of x[1 . . m] to see if it is also a subsequence of y[1 . . n].
Introduction to Algorithms October 25, 2004 L12.7© 2001–4 by Charles E. Leiserson
Brute-force LCS algorithm
Check every subsequence of x[1 . . m] to see if it is also a subsequence of y[1 . . n].
Analysis• Checking = O(n) time per subsequence.• 2m subsequences of x (each bit-vector of
length m determines a distinct subsequence of x).
Worst-case running time = O(n2m)= exponential time.
Introduction to Algorithms October 25, 2004 L12.8© 2001–4 by Charles E. Leiserson
Towards a better algorithmSimplification:1. Look at the length of a longest-common
subsequence. 2. Extend the algorithm to find the LCS itself.
Introduction to Algorithms October 25, 2004 L12.9© 2001–4 by Charles E. Leiserson
Towards a better algorithmSimplification:1. Look at the length of a longest-common
subsequence. 2. Extend the algorithm to find the LCS itself.
Notation: Denote the length of a sequence sby | s |.
Introduction to Algorithms October 25, 2004 L12.10© 2001–4 by Charles E. Leiserson
Towards a better algorithmSimplification:1. Look at the length of a longest-common
subsequence. 2. Extend the algorithm to find the LCS itself.
Strategy: Consider prefixes of x and y.• Define c[i, j] = | LCS(x[1 . . i], y[1 . . j]) |.• Then, c[m, n] = | LCS(x, y) |.
Notation: Denote the length of a sequence sby | s |.
Introduction to Algorithms October 25, 2004 L12.11© 2001–4 by Charles E. Leiserson
Recursive formulationTheorem.
c[i, j] =c[i–1, j–1] + 1 if x[i] = y[j],maxc[i–1, j], c[i, j–1] otherwise.
Introduction to Algorithms October 25, 2004 L12.12© 2001–4 by Charles E. Leiserson
Recursive formulationTheorem.
c[i, j] =c[i–1, j–1] + 1 if x[i] = y[j],maxc[i–1, j], c[i, j–1] otherwise.
Proof. Case x[i] = y[ j]:
L1 2 i m
L1 2 j n
x:
y:=
Introduction to Algorithms October 25, 2004 L12.13© 2001–4 by Charles E. Leiserson
Recursive formulationTheorem.
c[i, j] =c[i–1, j–1] + 1 if x[i] = y[j],maxc[i–1, j], c[i, j–1] otherwise.
Let z[1 . . k] = LCS(x[1 . . i], y[1 . . j]), where c[i, j] = k. Then, z[k] = x[i], or else z could be extended. Thus, z[1 . . k–1] is CS of x[1 . . i–1] and y[1 . . j–1].
Proof. Case x[i] = y[ j]:
L1 2 i m
L1 2 j n
x:
y:=
Introduction to Algorithms October 25, 2004 L12.14© 2001–4 by Charles E. Leiserson
Proof (continued)Claim: z[1 . . k–1] = LCS(x[1 . . i–1], y[1 . . j–1]).
Suppose w is a longer CS of x[1 . . i–1] andy[1 . . j–1], that is, |w | > k–1. Then, cut and paste: w || z[k] (w concatenated with z[k]) is a common subsequence of x[1 . . i] and y[1 . . j]with |w || z[k] | > k. Contradiction, proving the claim.
Introduction to Algorithms October 25, 2004 L12.15© 2001–4 by Charles E. Leiserson
Proof (continued)Claim: z[1 . . k–1] = LCS(x[1 . . i–1], y[1 . . j–1]).
Suppose w is a longer CS of x[1 . . i–1] andy[1 . . j–1], that is, |w | > k–1. Then, cut and paste: w || z[k] (w concatenated with z[k]) is a common subsequence of x[1 . . i] and y[1 . . j]with |w || z[k] | > k. Contradiction, proving the claim.
Thus, c[i–1, j–1] = k–1, which implies that c[i, j] = c[i–1, j–1] + 1.Other cases are similar.
Introduction to Algorithms October 25, 2004 L12.16© 2001–4 by Charles E. Leiserson
Dynamic-programming hallmark #1
Optimal substructureAn optimal solution to a problem
(instance) contains optimal solutions to subproblems.
Introduction to Algorithms October 25, 2004 L12.17© 2001–4 by Charles E. Leiserson
Dynamic-programming hallmark #1
Optimal substructureAn optimal solution to a problem
(instance) contains optimal solutions to subproblems.
If z = LCS(x, y), then any prefix of z is an LCS of a prefix of x and a prefix of y.
Introduction to Algorithms October 25, 2004 L12.18© 2001–4 by Charles E. Leiserson
Recursive algorithm for LCS
LCS(x, y, i, j)if x[i] = y[ j]
then c[i, j] ← LCS(x, y, i–1, j–1) + 1else c[i, j] ← maxLCS(x, y, i–1, j),
LCS(x, y, i, j–1)
Introduction to Algorithms October 25, 2004 L12.19© 2001–4 by Charles E. Leiserson
Recursive algorithm for LCS
LCS(x, y, i, j)if x[i] = y[ j]
then c[i, j] ← LCS(x, y, i–1, j–1) + 1else c[i, j] ← maxLCS(x, y, i–1, j),
LCS(x, y, i, j–1)Worst-case: x[i] ≠ y[ j], in which case the algorithm evaluates two subproblems, each with only one parameter decremented.
Introduction to Algorithms October 25, 2004 L12.20© 2001–4 by Charles E. Leiserson
Recursion treem = 3, n = 4: 3,43,4
2,42,4
1,41,4
3,33,3
3,23,22,32,3
1,31,3 2,22,2
2,32,3
1,31,3 2,22,2
Introduction to Algorithms October 25, 2004 L12.21© 2001–4 by Charles E. Leiserson
Recursion treem = 3, n = 4: 3,43,4
2,42,4
1,41,4
3,33,3
3,23,22,32,3
1,31,3 2,22,2
Height = m + n ⇒ work potentially exponential.
2,32,3
1,31,3 2,22,2
m+n
Introduction to Algorithms October 25, 2004 L12.22© 2001–4 by Charles E. Leiserson
same subproblem
,but we’re solving subproblems already solved!
Recursion treem = 3, n = 4: 3,43,4
2,42,4
1,41,4
3,33,3
3,23,22,32,3
1,31,3 2,22,2
Height = m + n ⇒ work potentially exponential.
2,32,3
1,31,3 2,22,2
m+n
Introduction to Algorithms October 25, 2004 L12.23© 2001–4 by Charles E. Leiserson
Dynamic-programming hallmark #2
Overlapping subproblemsA recursive solution contains a
“small” number of distinct subproblems repeated many times.
Introduction to Algorithms October 25, 2004 L12.24© 2001–4 by Charles E. Leiserson
Dynamic-programming hallmark #2
Overlapping subproblemsA recursive solution contains a
“small” number of distinct subproblems repeated many times.
The number of distinct LCS subproblems for two strings of lengths m and n is only mn.
Introduction to Algorithms October 25, 2004 L12.25© 2001–4 by Charles E. Leiserson
Memoization algorithmMemoization: After computing a solution to a subproblem, store it in a table. Subsequent calls check the table to avoid redoing work.
Introduction to Algorithms October 25, 2004 L12.26© 2001–4 by Charles E. Leiserson
Memoization algorithmMemoization: After computing a solution to a subproblem, store it in a table. Subsequent calls check the table to avoid redoing work.LCS(x, y, i, j)
if c[i, j] = NILthen if x[i] = y[j]
then c[i, j] ← LCS(x, y, i–1, j–1) + 1else c[i, j] ← maxLCS(x, y, i–1, j),
LCS(x, y, i, j–1)
same as before
Introduction to Algorithms October 25, 2004 L12.27© 2001–4 by Charles E. Leiserson
Memoization algorithmMemoization: After computing a solution to a subproblem, store it in a table. Subsequent calls check the table to avoid redoing work.
Time = Θ(mn) = constant work per table entry.Space = Θ(mn).
LCS(x, y, i, j)if c[i, j] = NIL
then if x[i] = y[j]then c[i, j] ← LCS(x, y, i–1, j–1) + 1else c[i, j] ← maxLCS(x, y, i–1, j),
LCS(x, y, i, j–1)
same as before
Introduction to Algorithms October 25, 2004 L12.28© 2001–4 by Charles E. Leiserson
00 00 00 00 00
00 00 11 11 1100 00 00
11 11 11
00 00 11 11 11 22 22D 22
00 00 11 22 22 22 22C 22
00 11 11 22 22 22 33A 33
00 11 22 22 33 33 33B 44
00 11 22 22 33 33
A
Dynamic-programming algorithm
IDEA:Compute the table bottom-up.
A B C B D B
B
A 44 44
Introduction to Algorithms October 25, 2004 L12.29© 2001–4 by Charles E. Leiserson
00 00 00 00 00
00 00 11 11 1100 00 00
11 11 11
00 00 11 11 11 22 22D 22
00 00 11 22 22 22 22C 22
00 11 11 22 22 22 33A 33
00 11 22 22 33 33 33B 44
00 11 22 22 33 33
A
Dynamic-programming algorithm
IDEA:Compute the table bottom-up.
A B C B D B
B
A 44 44
Time = Θ(mn).
Introduction to Algorithms October 25, 2004 L12.30© 2001–4 by Charles E. Leiserson
00 00 00 00 00 00 00 00
00 00 11 11 11 11 11 11
00 00 11 11 11 22 22D 22
00 00 11 22 22 22 22C 22
00 11 11 22 22 22 33A 33
00 11 22 22 33 33 33B 44
00 11 22 22 33 33
A
Dynamic-programming algorithm
IDEA:Compute the table bottom-up.
A B C B D B
B
A 44 44
Time = Θ(mn).Reconstruct LCS by tracing backwards.
0A
4
0B
B1
C
C
2B
B
3
A
A
D1
A2
D
3
B
4
Introduction to Algorithms October 25, 2004 L12.31© 2001–4 by Charles E. Leiserson
00 00 00 00 00 00 00 00
00 00 11 11 11 11 11 11
00 00 11 11 11 22 22D 22
00 00 11 22 22 22 22C 22
00 11 11 22 22 22 33A 33
00 11 22 22 33 33 33B 44
00 11 22 22 33 33
A
Dynamic-programming algorithm
IDEA:Compute the table bottom-up.
A B C B D B
B
A 44 44
Time = Θ(mn).Reconstruct LCS by tracing backwards.
0A
4
0B
B1
C
C
2B
B
3
A
A
D1
A2
D
3
B
4Space = Θ(mn).Exercise:O(minm, n).
Introduction to Algorithms6.046J/18.401J
Prof. Charles E. Leiserson
LECTURE 13Graph algorithms• Graph representation• Minimum spanning treesGreedy algorithms• Optimal substructure• Greedy choice• Prim’s greedy MST
algorithm
Introduction to Algorithms October 27, 2004 L13.2© 2001–4 by Charles E. Leiserson
Graphs (review)Definition. A directed graph (digraph)G = (V, E) is an ordered pair consisting of• a set V of vertices (singular: vertex),• a set E ⊆ V × V of edges.In an undirected graph G = (V, E), the edge set E consists of unordered pairs of vertices.In either case, we have |E | = O(V 2). Moreover, if G is connected, then |E | ≥ |V | – 1, which implies that lg |E | = Θ(lgV). (Review CLRS, Appendix B.)
Introduction to Algorithms October 27, 2004 L13.3© 2001–4 by Charles E. Leiserson
Adjacency-matrix representation
The adjacency matrix of a graph G = (V, E), where V = 1, 2, …, n, is the matrix A[1 . . n, 1 . . n]given by
A[i, j] = 1 if (i, j) ∈ E,0 if (i, j) ∉ E.
Introduction to Algorithms October 27, 2004 L13.4© 2001–4 by Charles E. Leiserson
Adjacency-matrix representation
The adjacency matrix of a graph G = (V, E), where V = 1, 2, …, n, is the matrix A[1 . . n, 1 . . n]given by
A[i, j] = 1 if (i, j) ∈ E,0 if (i, j) ∉ E.
22 11
33 44
A 1 2 3 41234
0 1 1 00 0 1 00 0 0 00 0 1 0
Θ(V 2) storage ⇒ denserepresentation.
Introduction to Algorithms October 27, 2004 L13.5© 2001–4 by Charles E. Leiserson
Adjacency-list representationAn adjacency list of a vertex v ∈ V is the list Adj[v]of vertices adjacent to v.
22 11
33 44
Adj[1] = 2, 3Adj[2] = 3Adj[3] = Adj[4] = 3
Introduction to Algorithms October 27, 2004 L13.6© 2001–4 by Charles E. Leiserson
Adjacency-list representationAn adjacency list of a vertex v ∈ V is the list Adj[v]of vertices adjacent to v.
22 11
33 44
Adj[1] = 2, 3Adj[2] = 3Adj[3] = Adj[4] = 3
For undirected graphs, |Adj[v] | = degree(v).For digraphs, |Adj[v] | = out-degree(v).
Introduction to Algorithms October 27, 2004 L13.7© 2001–4 by Charles E. Leiserson
Adjacency-list representationAn adjacency list of a vertex v ∈ V is the list Adj[v]of vertices adjacent to v.
22 11
33 44
Adj[1] = 2, 3Adj[2] = 3Adj[3] = Adj[4] = 3
For undirected graphs, |Adj[v] | = degree(v).For digraphs, |Adj[v] | = out-degree(v).Handshaking Lemma: ∑v∈V = 2 |E | for undirected graphs ⇒ adjacency lists use Θ(V + E) storage —a sparse representation (for either type of graph).
Introduction to Algorithms October 27, 2004 L13.8© 2001–4 by Charles E. Leiserson
Minimum spanning trees
Input: A connected, undirected graph G = (V, E)with weight function w : E → R.• For simplicity, assume that all edge weights are
distinct. (CLRS covers the general case.)
Introduction to Algorithms October 27, 2004 L13.9© 2001–4 by Charles E. Leiserson
Minimum spanning trees
Input: A connected, undirected graph G = (V, E)with weight function w : E → R.• For simplicity, assume that all edge weights are
distinct. (CLRS covers the general case.)
∑∈
=Tvu
vuwTw),(
),()( .
Output: A spanning tree T — a tree that connects all vertices — of minimum weight:
Introduction to Algorithms October 27, 2004 L13.10© 2001–4 by Charles E. Leiserson
Example of MST
6 125
14
3
8
10
15
9
7
Introduction to Algorithms October 27, 2004 L13.11© 2001–4 by Charles E. Leiserson
Example of MST
6 125
14
3
8
10
15
9
7
Introduction to Algorithms October 27, 2004 L13.12© 2001–4 by Charles E. Leiserson
Optimal substructureMST T:
(Other edges of Gare not shown.)
Introduction to Algorithms October 27, 2004 L13.13© 2001–4 by Charles E. Leiserson
u
vRemove any edge (u, v) ∈ T.
Optimal substructureMST T:
(Other edges of Gare not shown.)
Introduction to Algorithms October 27, 2004 L13.14© 2001–4 by Charles E. Leiserson
u
vRemove any edge (u, v) ∈ T.
Optimal substructureMST T:
(Other edges of Gare not shown.)
Introduction to Algorithms October 27, 2004 L13.15© 2001–4 by Charles E. Leiserson
u
vRemove any edge (u, v) ∈ T. Remove any edge (u, v) ∈ T. Then, T is partitioned into two subtrees T1 and T2.
T1
T2u
v
Optimal substructureMST T:
(Other edges of Gare not shown.)
Introduction to Algorithms October 27, 2004 L13.16© 2001–4 by Charles E. Leiserson
u
vRemove any edge (u, v) ∈ T. Remove any edge (u, v) ∈ T. Then, T is partitioned into two subtrees T1 and T2.
T1
T2u
v
Optimal substructureMST T:
(Other edges of Gare not shown.)
Theorem. The subtree T1 is an MST of G1 = (V1, E1), the subgraph of G induced by the vertices of T1:
V1 = vertices of T1,E1 = (x, y) ∈ E : x, y ∈ V1 .
Similarly for T2.
Introduction to Algorithms October 27, 2004 L13.17© 2001–4 by Charles E. Leiserson
Proof of optimal substructure
w(T) = w(u, v) + w(T1) + w(T2).Proof. Cut and paste:
If T1′were a lower-weight spanning tree than T1 for G1, then T ′ = (u, v) ∪ T1′ ∪ T2 would be a lower-weight spanning tree than T for G.
Introduction to Algorithms October 27, 2004 L13.18© 2001–4 by Charles E. Leiserson
Proof of optimal substructure
w(T) = w(u, v) + w(T1) + w(T2).Proof. Cut and paste:
If T1′were a lower-weight spanning tree than T1 for G1, then T ′ = (u, v) ∪ T1′ ∪ T2 would be a lower-weight spanning tree than T for G.
Do we also have overlapping subproblems?•Yes.
Introduction to Algorithms October 27, 2004 L13.19© 2001–4 by Charles E. Leiserson
Proof of optimal substructure
w(T) = w(u, v) + w(T1) + w(T2).Proof. Cut and paste:
If T1′were a lower-weight spanning tree than T1 for G1, then T ′ = (u, v) ∪ T1′ ∪ T2 would be a lower-weight spanning tree than T for G.
Great, then dynamic programming may work!•Yes, but MST exhibits another powerful property which leads to an even more efficient algorithm.
Do we also have overlapping subproblems?•Yes.
Introduction to Algorithms October 27, 2004 L13.20© 2001–4 by Charles E. Leiserson
Hallmark for “greedy” algorithms
Greedy-choice propertyA locally optimal choice
is globally optimal.
Introduction to Algorithms October 27, 2004 L13.21© 2001–4 by Charles E. Leiserson
Hallmark for “greedy” algorithms
Greedy-choice propertyA locally optimal choice
is globally optimal.
Theorem. Let T be the MST of G = (V, E), and let A ⊆ V. Suppose that (u, v) ∈ E is the least-weight edge connecting A to V – A. Then, (u, v) ∈ T.
Introduction to Algorithms October 27, 2004 L13.22© 2001–4 by Charles E. Leiserson
Proof of theoremProof. Suppose (u, v) ∉ T. Cut and paste.
∈ A∈ V – A
T:
u
v
(u, v) = least-weight edge connecting A to V – A
Introduction to Algorithms October 27, 2004 L13.23© 2001–4 by Charles E. Leiserson
Proof of theoremProof. Suppose (u, v) ∉ T. Cut and paste.
∈ A∈ V – A
T:
u
Consider the unique simple path from u to v in T.
(u, v) = least-weight edge connecting A to V – A
v
Introduction to Algorithms October 27, 2004 L13.24© 2001–4 by Charles E. Leiserson
Proof of theoremProof. Suppose (u, v) ∉ T. Cut and paste.
∈ A∈ V – A
T:
u(u, v) = least-weight edge connecting A to V – A
v
Consider the unique simple path from u to v in T. Swap (u, v) with the first edge on this path that connects a vertex in A to a vertex in V – A.
Introduction to Algorithms October 27, 2004 L13.25© 2001–4 by Charles E. Leiserson
Proof of theoremProof. Suppose (u, v) ∉ T. Cut and paste.
∈ A∈ V – A
T ′:
u(u, v) = least-weight edge connecting A to V – A
v
Consider the unique simple path from u to v in T. Swap (u, v) with the first edge on this path that connects a vertex in A to a vertex in V – A.A lighter-weight spanning tree than T results.
Introduction to Algorithms October 27, 2004 L13.26© 2001–4 by Charles E. Leiserson
Prim’s algorithmIDEA: Maintain V – A as a priority queue Q. Key each vertex in Q with the weight of the least-weight edge connecting it to a vertex in A.Q ← Vkey[v] ←∞ for all v ∈ Vkey[s] ← 0 for some arbitrary s ∈ Vwhile Q ≠ ∅
do u ← EXTRACT-MIN(Q)for each v ∈ Adj[u]
do if v ∈ Q and w(u, v) < key[v]then key[v] ← w(u, v) ⊳ DECREASE-KEY
π[v] ← u
At the end, (v, π[v]) forms the MST.
Introduction to Algorithms October 27, 2004 L13.27© 2001–4 by Charles E. Leiserson
Example of Prim’s algorithm
∈ A∈ V – A
∞∞
∞∞ ∞∞
∞∞ 00
∞∞
∞∞
∞∞
6 125
14
3
8
10
15
9
7
Introduction to Algorithms October 27, 2004 L13.28© 2001–4 by Charles E. Leiserson
Example of Prim’s algorithm
∈ A∈ V – A
∞∞
∞∞ ∞∞
∞∞ 00
∞∞
∞∞
∞∞
6 125
14
3
8
10
15
9
7
Introduction to Algorithms October 27, 2004 L13.29© 2001–4 by Charles E. Leiserson
Example of Prim’s algorithm
∈ A∈ V – A
∞∞
∞∞ 77
∞∞ 00
1010
∞∞
1515
6 125
14
3
8
10
15
9
7
Introduction to Algorithms October 27, 2004 L13.30© 2001–4 by Charles E. Leiserson
Example of Prim’s algorithm
∈ A∈ V – A
∞∞
∞∞ 77
∞∞ 00
1010
∞∞
1515
6 125
14
3
8
10
15
9
7
Introduction to Algorithms October 27, 2004 L13.31© 2001–4 by Charles E. Leiserson
Example of Prim’s algorithm
∈ A∈ V – A
1212
55 77
∞∞ 00
1010
99
1515
6 125
14
3
8
10
15
9
7
Introduction to Algorithms October 27, 2004 L13.32© 2001–4 by Charles E. Leiserson
Example of Prim’s algorithm
∈ A∈ V – A
1212
55 77
∞∞ 00
1010
99
1515
6 125
14
3
8
10
15
9
7
Introduction to Algorithms October 27, 2004 L13.33© 2001–4 by Charles E. Leiserson
Example of Prim’s algorithm
∈ A∈ V – A
66
55 77
1414 00
88
99
1515
6 125
14
3
8
10
15
9
7
Introduction to Algorithms October 27, 2004 L13.34© 2001–4 by Charles E. Leiserson
Example of Prim’s algorithm
∈ A∈ V – A
66
55 77
1414 00
88
99
1515
6 125
14
3
8
10
15
9
7
Introduction to Algorithms October 27, 2004 L13.35© 2001–4 by Charles E. Leiserson
Example of Prim’s algorithm
∈ A∈ V – A
66
55 77
1414 00
88
99
1515
6 125
14
3
8
10
15
9
7
Introduction to Algorithms October 27, 2004 L13.36© 2001–4 by Charles E. Leiserson
Example of Prim’s algorithm
∈ A∈ V – A
66
55 77
33 00
88
99
1515
6 125
14
3
8
10
15
9
7
Introduction to Algorithms October 27, 2004 L13.37© 2001–4 by Charles E. Leiserson
Example of Prim’s algorithm
∈ A∈ V – A
66
55 77
33 00
88
99
1515
6 125
14
3
8
10
15
9
7
Introduction to Algorithms October 27, 2004 L13.38© 2001–4 by Charles E. Leiserson
Example of Prim’s algorithm
∈ A∈ V – A
66
55 77
33 00
88
99
1515
6 125
14
3
8
10
15
9
7
Introduction to Algorithms October 27, 2004 L13.39© 2001–4 by Charles E. Leiserson
Example of Prim’s algorithm
∈ A∈ V – A
66
55 77
33 00
88
99
1515
6 125
14
3
8
10
15
9
7
Introduction to Algorithms October 27, 2004 L13.40© 2001–4 by Charles E. Leiserson
Q ← Vkey[v] ←∞ for all v ∈ Vkey[s] ← 0 for some arbitrary s ∈ Vwhile Q ≠ ∅
do u ← EXTRACT-MIN(Q)for each v ∈ Adj[u]
do if v ∈ Q and w(u, v) < key[v]then key[v] ← w(u, v)
π[v] ← u
Analysis of Prim
Introduction to Algorithms October 27, 2004 L13.41© 2001–4 by Charles E. Leiserson
Q ← Vkey[v] ←∞ for all v ∈ Vkey[s] ← 0 for some arbitrary s ∈ Vwhile Q ≠ ∅
do u ← EXTRACT-MIN(Q)for each v ∈ Adj[u]
do if v ∈ Q and w(u, v) < key[v]then key[v] ← w(u, v)
π[v] ← u
Analysis of Prim
Θ(V)total
Introduction to Algorithms October 27, 2004 L13.42© 2001–4 by Charles E. Leiserson
Q ← Vkey[v] ←∞ for all v ∈ Vkey[s] ← 0 for some arbitrary s ∈ Vwhile Q ≠ ∅
do u ← EXTRACT-MIN(Q)for each v ∈ Adj[u]
do if v ∈ Q and w(u, v) < key[v]then key[v] ← w(u, v)
π[v] ← u
Analysis of Prim
|V |times
Θ(V)total
Introduction to Algorithms October 27, 2004 L13.43© 2001–4 by Charles E. Leiserson
Q ← Vkey[v] ←∞ for all v ∈ Vkey[s] ← 0 for some arbitrary s ∈ Vwhile Q ≠ ∅
do u ← EXTRACT-MIN(Q)for each v ∈ Adj[u]
do if v ∈ Q and w(u, v) < key[v]then key[v] ← w(u, v)
π[v] ← u
Analysis of Prim
degree(u)times
|V |times
Θ(V)total
Introduction to Algorithms October 27, 2004 L13.44© 2001–4 by Charles E. Leiserson
Handshaking Lemma ⇒Θ(E) implicit DECREASE-KEY’s.
Q ← Vkey[v] ←∞ for all v ∈ Vkey[s] ← 0 for some arbitrary s ∈ Vwhile Q ≠ ∅
do u ← EXTRACT-MIN(Q)for each v ∈ Adj[u]
do if v ∈ Q and w(u, v) < key[v]then key[v] ← w(u, v)
π[v] ← u
Analysis of Prim
degree(u)times
|V |times
Θ(V)total
Introduction to Algorithms October 27, 2004 L13.45© 2001–4 by Charles E. Leiserson
Handshaking Lemma ⇒Θ(E) implicit DECREASE-KEY’s.
Q ← Vkey[v] ←∞ for all v ∈ Vkey[s] ← 0 for some arbitrary s ∈ Vwhile Q ≠ ∅
do u ← EXTRACT-MIN(Q)for each v ∈ Adj[u]
do if v ∈ Q and w(u, v) < key[v]then key[v] ← w(u, v)
π[v] ← u
Analysis of Prim
degree(u)times
|V |times
Θ(V)total
Time = Θ(V)·TEXTRACT-MIN + Θ(E)·TDECREASE-KEY
Introduction to Algorithms October 27, 2004 L13.46© 2001–4 by Charles E. Leiserson
Analysis of Prim (continued)
Time = Θ(V)·TEXTRACT-MIN + Θ(E)·TDECREASE-KEY
Introduction to Algorithms October 27, 2004 L13.47© 2001–4 by Charles E. Leiserson
Analysis of Prim (continued)
Time = Θ(V)·TEXTRACT-MIN + Θ(E)·TDECREASE-KEY
Q TEXTRACT-MIN TDECREASE-KEY Total
Introduction to Algorithms October 27, 2004 L13.48© 2001–4 by Charles E. Leiserson
Analysis of Prim (continued)
Time = Θ(V)·TEXTRACT-MIN + Θ(E)·TDECREASE-KEY
Q TEXTRACT-MIN TDECREASE-KEY Total
array O(V) O(1) O(V2)
Introduction to Algorithms October 27, 2004 L13.49© 2001–4 by Charles E. Leiserson
Analysis of Prim (continued)
Time = Θ(V)·TEXTRACT-MIN + Θ(E)·TDECREASE-KEY
Q TEXTRACT-MIN TDECREASE-KEY Total
array O(V) O(1) O(V2)binary heap O(lg V) O(lg V) O(E lg V)
Introduction to Algorithms October 27, 2004 L13.50© 2001–4 by Charles E. Leiserson
Analysis of Prim (continued)
Time = Θ(V)·TEXTRACT-MIN + Θ(E)·TDECREASE-KEY
Q TEXTRACT-MIN TDECREASE-KEY Total
array O(V) O(1) O(V2)binary heap O(lg V) O(lg V) O(E lg V)
Fibonacci heap
O(lg V)amortized
O(1)amortized
O(E + V lg V)worst case
Introduction to Algorithms October 27, 2004 L13.51© 2001–4 by Charles E. Leiserson
MST algorithms
Kruskal’s algorithm (see CLRS):• Uses the disjoint-set data structure (Lecture 10).• Running time = O(E lg V).
Introduction to Algorithms October 27, 2004 L13.52© 2001–4 by Charles E. Leiserson
MST algorithms
Kruskal’s algorithm (see CLRS):• Uses the disjoint-set data structure (Lecture 10).• Running time = O(E lg V).
Best to date:• Karger, Klein, and Tarjan [1993].• Randomized algorithm.• O(V + E) expected time.
Introduction to Algorithms6.046J/18.401J
Prof. Charles E. Leiserson
LECTURE 14Shortest Paths I• Properties of shortest paths• Dijkstra’s algorithm• Correctness• Analysis• Breadth-first search
Introduction to Algorithms November 1, 2004 L14.2© 2001–4 by Charles E. Leiserson
Paths in graphsConsider a digraph G = (V, E) with edge-weight function w : E → R. The weight of path p = v1 →v2 →L→ vk is defined to be
∑−
=+=
1
11),()(
k
iii vvwpw .
Introduction to Algorithms November 1, 2004 L14.3© 2001–4 by Charles E. Leiserson
Paths in graphsConsider a digraph G = (V, E) with edge-weight function w : E → R. The weight of path p = v1 →v2 →L→ vk is defined to be
∑−
=+=
1
11),()(
k
iii vvwpw .
v1v1
v2v2
v3v3
v4v4
v5v54 –2 –5 1
Example:
w(p) = –2
Introduction to Algorithms November 1, 2004 L14.4© 2001–4 by Charles E. Leiserson
Shortest paths
A shortest path from u to v is a path of minimum weight from u to v. The shortest-path weight from u to v is defined asδ(u, v) = minw(p) : p is a path from u to v.
Note: δ(u, v) = ∞ if no path from u to v exists.
Introduction to Algorithms November 1, 2004 L14.5© 2001–4 by Charles E. Leiserson
Optimal substructure
Theorem. A subpath of a shortest path is a shortest path.
Introduction to Algorithms November 1, 2004 L14.6© 2001–4 by Charles E. Leiserson
Optimal substructure
Theorem. A subpath of a shortest path is a shortest path.
Proof. Cut and paste:
Introduction to Algorithms November 1, 2004 L14.7© 2001–4 by Charles E. Leiserson
Optimal substructure
Theorem. A subpath of a shortest path is a shortest path.
Proof. Cut and paste:
Introduction to Algorithms November 1, 2004 L14.8© 2001–4 by Charles E. Leiserson
Triangle inequality
Theorem. For all u, v, x ∈ V, we haveδ(u, v) ≤ δ(u, x) + δ(x, v).
Introduction to Algorithms November 1, 2004 L14.9© 2001–4 by Charles E. Leiserson
Triangle inequality
Theorem. For all u, v, x ∈ V, we haveδ(u, v) ≤ δ(u, x) + δ(x, v).
uu
Proof.
xx
vvδ(u, v)
δ(u, x) δ(x, v)
Introduction to Algorithms November 1, 2004 L14.10© 2001–4 by Charles E. Leiserson
Well-definedness of shortest paths
If a graph G contains a negative-weight cycle, then some shortest paths may not exist.
Introduction to Algorithms November 1, 2004 L14.11© 2001–4 by Charles E. Leiserson
Well-definedness of shortest paths
If a graph G contains a negative-weight cycle, then some shortest paths may not exist.
Example:
uu vv
…
< 0
Introduction to Algorithms November 1, 2004 L14.12© 2001–4 by Charles E. Leiserson
Single-source shortest pathsProblem. From a given source vertex s ∈ V, find the shortest-path weights δ(s, v) for all v ∈ V.If all edge weights w(u, v) are nonnegative, all shortest-path weights must exist. IDEA: Greedy.1. Maintain a set S of vertices whose shortest-
path distances from s are known.2. At each step add to S the vertex v ∈ V – S
whose distance estimate from s is minimal.3. Update the distance estimates of vertices
adjacent to v.
Introduction to Algorithms November 1, 2004 L14.13© 2001–4 by Charles E. Leiserson
Dijkstra’s algorithmd[s] ← 0for each v ∈ V – s
do d[v] ←∞S ←∅Q ← V ⊳ Q is a priority queue maintaining V – S
Introduction to Algorithms November 1, 2004 L14.14© 2001–4 by Charles E. Leiserson
Dijkstra’s algorithmd[s] ← 0for each v ∈ V – s
do d[v] ←∞S ←∅Q ← V ⊳ Q is a priority queue maintaining V – Swhile Q ≠ ∅
do u ← EXTRACT-MIN(Q)S ← S ∪ ufor each v ∈ Adj[u]
do if d[v] > d[u] + w(u, v)then d[v] ← d[u] + w(u, v)
Introduction to Algorithms November 1, 2004 L14.15© 2001–4 by Charles E. Leiserson
Dijkstra’s algorithmd[s] ← 0for each v ∈ V – s
do d[v] ←∞S ←∅Q ← V ⊳ Q is a priority queue maintaining V – Swhile Q ≠ ∅
do u ← EXTRACT-MIN(Q)S ← S ∪ ufor each v ∈ Adj[u]
do if d[v] > d[u] + w(u, v)then d[v] ← d[u] + w(u, v)
relaxation step
Implicit DECREASE-KEY
Introduction to Algorithms November 1, 2004 L14.16© 2001–4 by Charles E. Leiserson
Example of Dijkstra’s algorithm
AA
BB DD
CC EE
10
3
1 4 7 98
2
2
Graph with nonnegative edge weights:
Introduction to Algorithms November 1, 2004 L14.17© 2001–4 by Charles E. Leiserson
Example of Dijkstra’s algorithm
AA
BB DD
CC EE
10
3
1 4 7 98
2
2
Initialize:
A B C D EQ:0 ∞ ∞ ∞ ∞
S:
0
∞
∞ ∞
∞
Introduction to Algorithms November 1, 2004 L14.18© 2001–4 by Charles E. Leiserson
Example of Dijkstra’s algorithm
AA
BB DD
CC EE
10
3
1 4 7 98
2
2A B C D EQ:0 ∞ ∞ ∞ ∞
S: A
0
∞
∞ ∞
∞“A” ← EXTRACT-MIN(Q):
Introduction to Algorithms November 1, 2004 L14.19© 2001–4 by Charles E. Leiserson
Example of Dijkstra’s algorithm
AA
BB DD
CC EE
10
3
1 4 7 98
2
2A B C D EQ:0 ∞ ∞ ∞ ∞
S: A
0
10
3 ∞
∞
10 3
Relax all edges leaving A:
∞ ∞
Introduction to Algorithms November 1, 2004 L14.20© 2001–4 by Charles E. Leiserson
Example of Dijkstra’s algorithm
AA
BB DD
CC EE
10
3
1 4 7 98
2
2A B C D EQ:0 ∞ ∞ ∞ ∞
S: A, C
0
10
3 ∞
∞
10 3
“C” ← EXTRACT-MIN(Q):
∞ ∞
Introduction to Algorithms November 1, 2004 L14.21© 2001–4 by Charles E. Leiserson
Example of Dijkstra’s algorithm
AA
BB DD
CC EE
10
3
1 4 7 98
2
2A B C D EQ:0 ∞ ∞ ∞ ∞
S: A, C
0
7
3 5
11
10 37 11 5
Relax all edges leaving C:
∞ ∞
Introduction to Algorithms November 1, 2004 L14.22© 2001–4 by Charles E. Leiserson
Example of Dijkstra’s algorithm
AA
BB DD
CC EE
10
3
1 4 7 98
2
2A B C D EQ:0 ∞ ∞ ∞ ∞
S: A, C, E
0
7
3 5
11
10 37 11 5
“E” ← EXTRACT-MIN(Q):
∞ ∞
Introduction to Algorithms November 1, 2004 L14.23© 2001–4 by Charles E. Leiserson
Example of Dijkstra’s algorithm
AA
BB DD
CC EE
10
3
1 4 7 98
2
2A B C D EQ:0 ∞ ∞ ∞ ∞
S: A, C, E
0
7
3 5
11
10 3 ∞ ∞7 11 57 11
Relax all edges leaving E:
Introduction to Algorithms November 1, 2004 L14.24© 2001–4 by Charles E. Leiserson
Example of Dijkstra’s algorithm
AA
BB DD
CC EE
10
3
1 4 7 98
2
2A B C D EQ:0 ∞ ∞ ∞ ∞
S: A, C, E, B
0
7
3 5
11
10 3 ∞ ∞7 11 57 11
“B” ← EXTRACT-MIN(Q):
Introduction to Algorithms November 1, 2004 L14.25© 2001–4 by Charles E. Leiserson
Example of Dijkstra’s algorithm
AA
BB DD
CC EE
10
3
1 4 7 98
2
2A B C D EQ:0 ∞ ∞ ∞ ∞
S: A, C, E, B
0
7
3 5
9
10 3 ∞ ∞7 11 57 11
Relax all edges leaving B:
9
Introduction to Algorithms November 1, 2004 L14.26© 2001–4 by Charles E. Leiserson
Example of Dijkstra’s algorithm
AA
BB DD
CC EE
10
3
1 4 7 98
2
2A B C D EQ:0 ∞ ∞ ∞ ∞
S: A, C, E, B, D
0
7
3 5
9
10 3 ∞ ∞7 11 57 11
9
“D” ← EXTRACT-MIN(Q):
Introduction to Algorithms November 1, 2004 L14.27© 2001–4 by Charles E. Leiserson
Correctness — Part ILemma. Initializing d[s] ← 0 and d[v] ←∞ for all v ∈ V – s establishes d[v] ≥ δ(s, v) for all v ∈ V, and this invariant is maintained over any sequence of relaxation steps.
Introduction to Algorithms November 1, 2004 L14.28© 2001–4 by Charles E. Leiserson
Correctness — Part ILemma. Initializing d[s] ← 0 and d[v] ←∞ for all v ∈ V – s establishes d[v] ≥ δ(s, v) for all v ∈ V, and this invariant is maintained over any sequence of relaxation steps.Proof. Suppose not. Let v be the first vertex for which d[v] < δ(s, v), and let u be the vertex that caused d[v] to change: d[v] = d[u] + w(u, v). Then,
d[v] < δ(s, v) supposition≤ δ(s, u) + δ(u, v) triangle inequality≤ δ(s,u) + w(u, v) sh. path ≤ specific path≤ d[u] + w(u, v) v is first violation
Contradiction.
Introduction to Algorithms November 1, 2004 L14.29© 2001–4 by Charles E. Leiserson
Correctness — Part IILemma. Let u be v’s predecessor on a shortest path from s to v. Then, if d[u] = δ(s, u) and edge (u, v) is relaxed, we have d[v] = δ(s, v) after the relaxation.
Introduction to Algorithms November 1, 2004 L14.30© 2001–4 by Charles E. Leiserson
Correctness — Part IILemma. Let u be v’s predecessor on a shortest path from s to v. Then, if d[u] = δ(s, u) and edge (u, v) is relaxed, we have d[v] = δ(s, v) after the relaxation.Proof. Observe that δ(s, v) = δ(s, u) + w(u, v). Suppose that d[v] > δ(s, v) before the relaxation. (Otherwise, we’re done.) Then, the test d[v] > d[u] + w(u, v) succeeds, because d[v] > δ(s, v) = δ(s, u) + w(u, v) = d[u] + w(u, v), and the algorithm sets d[v] = d[u] + w(u, v) = δ(s, v).
Introduction to Algorithms November 1, 2004 L14.31© 2001–4 by Charles E. Leiserson
Correctness — Part IIITheorem. Dijkstra’s algorithm terminates with d[v] = δ(s, v) for all v ∈ V.
Introduction to Algorithms November 1, 2004 L14.32© 2001–4 by Charles E. Leiserson
Correctness — Part IIITheorem. Dijkstra’s algorithm terminates with d[v] = δ(s, v) for all v ∈ V.Proof. It suffices to show that d[v] = δ(s, v) for every v ∈ V when v is added to S. Suppose u is the first vertex added to S for which d[u] > δ(s, u). Let y be the first vertex in V – S along a shortest path from s to u, and let x be its predecessor:
ss xx yy
uu
S, just before adding u.
Introduction to Algorithms November 1, 2004 L14.33© 2001–4 by Charles E. Leiserson
Correctness — Part III (continued)
Since u is the first vertex violating the claimed invariant, we have d[x] = δ(s, x). When x was added to S, the edge (x, y) was relaxed, which implies that d[y] = δ(s, y) ≤ δ(s, u) < d[u]. But, d[u] ≤ d[y] by our choice of u. Contradiction.
ss xx yy
uuS
Introduction to Algorithms November 1, 2004 L14.34© 2001–4 by Charles E. Leiserson
Analysis of Dijkstrawhile Q ≠ ∅
do u ← EXTRACT-MIN(Q)S ← S ∪ ufor each v ∈ Adj[u]
do if d[v] > d[u] + w(u, v)then d[v] ← d[u] + w(u, v)
Introduction to Algorithms November 1, 2004 L14.35© 2001–4 by Charles E. Leiserson
Analysis of Dijkstra
|V |times
while Q ≠ ∅do u ← EXTRACT-MIN(Q)
S ← S ∪ ufor each v ∈ Adj[u]
do if d[v] > d[u] + w(u, v)then d[v] ← d[u] + w(u, v)
Introduction to Algorithms November 1, 2004 L14.36© 2001–4 by Charles E. Leiserson
Analysis of Dijkstra
degree(u)times
|V |times
while Q ≠ ∅do u ← EXTRACT-MIN(Q)
S ← S ∪ ufor each v ∈ Adj[u]
do if d[v] > d[u] + w(u, v)then d[v] ← d[u] + w(u, v)
Introduction to Algorithms November 1, 2004 L14.37© 2001–4 by Charles E. Leiserson
Analysis of Dijkstra
degree(u)times
|V |times
Handshaking Lemma ⇒Θ(E) implicit DECREASE-KEY’s.
while Q ≠ ∅do u ← EXTRACT-MIN(Q)
S ← S ∪ ufor each v ∈ Adj[u]
do if d[v] > d[u] + w(u, v)then d[v] ← d[u] + w(u, v)
Introduction to Algorithms November 1, 2004 L14.38© 2001–4 by Charles E. Leiserson
Analysis of Dijkstra
degree(u)times
|V |times
Handshaking Lemma ⇒Θ(E) implicit DECREASE-KEY’s.Time = Θ(V·TEXTRACT-MIN + E·TDECREASE-KEY)
Note: Same formula as in the analysis of Prim’s minimum spanning tree algorithm.
while Q ≠ ∅do u ← EXTRACT-MIN(Q)
S ← S ∪ ufor each v ∈ Adj[u]
do if d[v] > d[u] + w(u, v)then d[v] ← d[u] + w(u, v)
Introduction to Algorithms November 1, 2004 L14.39© 2001–4 by Charles E. Leiserson
Analysis of Dijkstra (continued)
Time = Θ(V)·TEXTRACT-MIN + Θ(E)·TDECREASE-KEY
Q TEXTRACT-MIN TDECREASE-KEY Total
Introduction to Algorithms November 1, 2004 L14.40© 2001–4 by Charles E. Leiserson
Analysis of Dijkstra (continued)
Time = Θ(V)·TEXTRACT-MIN + Θ(E)·TDECREASE-KEY
Q TEXTRACT-MIN TDECREASE-KEY Total
array O(V) O(1) O(V2)
Introduction to Algorithms November 1, 2004 L14.41© 2001–4 by Charles E. Leiserson
Analysis of Dijkstra (continued)
Time = Θ(V)·TEXTRACT-MIN + Θ(E)·TDECREASE-KEY
Q TEXTRACT-MIN TDECREASE-KEY Total
array O(V) O(1) O(V2)binary heap O(lg V) O(lg V) O(E lg V)
Introduction to Algorithms November 1, 2004 L14.42© 2001–4 by Charles E. Leiserson
Analysis of Dijkstra (continued)
Time = Θ(V)·TEXTRACT-MIN + Θ(E)·TDECREASE-KEY
Q TEXTRACT-MIN TDECREASE-KEY Total
array O(V) O(1) O(V2)binary heap O(lg V) O(lg V) O(E lg V)
Fibonacci heap
O(lg V)amortized
O(1)amortized
O(E + V lg V)worst case
Introduction to Algorithms November 1, 2004 L14.43© 2001–4 by Charles E. Leiserson
Unweighted graphsSuppose that w(u, v) = 1 for all (u, v) ∈ E. Can Dijkstra’s algorithm be improved?
Introduction to Algorithms November 1, 2004 L14.44© 2001–4 by Charles E. Leiserson
Unweighted graphs
• Use a simple FIFO queue instead of a priority queue.
Suppose that w(u, v) = 1 for all (u, v) ∈ E. Can Dijkstra’s algorithm be improved?
Introduction to Algorithms November 1, 2004 L14.45© 2001–4 by Charles E. Leiserson
Unweighted graphs
while Q ≠ ∅do u ← DEQUEUE(Q)
for each v ∈ Adj[u]do if d[v] = ∞
then d[v] ← d[u] + 1ENQUEUE(Q, v)
• Use a simple FIFO queue instead of a priority queue.
Breadth-first search
Suppose that w(u, v) = 1 for all (u, v) ∈ E. Can Dijkstra’s algorithm be improved?
Introduction to Algorithms November 1, 2004 L14.46© 2001–4 by Charles E. Leiserson
Unweighted graphs
while Q ≠ ∅do u ← DEQUEUE(Q)
for each v ∈ Adj[u]do if d[v] = ∞
then d[v] ← d[u] + 1ENQUEUE(Q, v)
• Use a simple FIFO queue instead of a priority queue.
Analysis: Time = O(V + E).
Breadth-first search
Suppose that w(u, v) = 1 for all (u, v) ∈ E. Can Dijkstra’s algorithm be improved?
Introduction to Algorithms November 1, 2004 L14.47© 2001–4 by Charles E. Leiserson
Example of breadth-first search
aa
bb
cc
dd
eegg
ii
ff hh
Q:
Introduction to Algorithms November 1, 2004 L14.48© 2001–4 by Charles E. Leiserson
Example of breadth-first search
aa
bb
cc
dd
eegg
ii
ff hh
Q: a
0
0
Introduction to Algorithms November 1, 2004 L14.49© 2001–4 by Charles E. Leiserson
Example of breadth-first search
aa
bb
cc
dd
eegg
ii
ff hh
Q: a b d
0
1
1
1 1
Introduction to Algorithms November 1, 2004 L14.50© 2001–4 by Charles E. Leiserson
Example of breadth-first search
aa
bb
cc
dd
eegg
ii
ff hh
Q: a b d c e
0
1
1
2 2
1 2 2
Introduction to Algorithms November 1, 2004 L14.51© 2001–4 by Charles E. Leiserson
Example of breadth-first search
aa
bb
cc
dd
eegg
ii
ff hh
Q: a b d c e
0
1
1
2 2
2 2
Introduction to Algorithms November 1, 2004 L14.52© 2001–4 by Charles E. Leiserson
Example of breadth-first search
aa
bb
cc
dd
eegg
ii
ff hh
Q: a b d c e
0
1
1
2 2
2
Introduction to Algorithms November 1, 2004 L14.53© 2001–4 by Charles E. Leiserson
Example of breadth-first search
aa
bb
cc
dd
eegg
ii
ff hh
Q: a b d c e g i
0
1
1
2 2
3
3
3 3
Introduction to Algorithms November 1, 2004 L14.54© 2001–4 by Charles E. Leiserson
Example of breadth-first search
aa
bb
cc
dd
eegg
ii
ff hh
Q: a b d c e g i f
0
1
1
2 2
3
3
4
3 4
Introduction to Algorithms November 1, 2004 L14.55© 2001–4 by Charles E. Leiserson
Example of breadth-first search
aa
bb
cc
dd
eegg
ii
ff hh
Q: a b d c e g i f h
0
1
1
2 2
3
3
4 4
4 4
Introduction to Algorithms November 1, 2004 L14.56© 2001–4 by Charles E. Leiserson
Example of breadth-first search
aa
bb
cc
dd
eegg
ii
ff hh
Q: a b d c e g i f h
0
1
1
2 2
3
3
4 4
4
Introduction to Algorithms November 1, 2004 L14.57© 2001–4 by Charles E. Leiserson
Example of breadth-first search
aa
bb
cc
dd
eegg
ii
ff hh
Q: a b d c e g i f h
0
1
1
2 2
3
3
4 4
Introduction to Algorithms November 1, 2004 L14.58© 2001–4 by Charles E. Leiserson
Example of breadth-first search
aa
bb
cc
dd
eegg
ii
ff hh
Q: a b d c e g i f h
0
1
1
2 2
3
3
4 4
Introduction to Algorithms November 1, 2004 L14.59© 2001–4 by Charles E. Leiserson
Correctness of BFS
Key idea:The FIFO Q in breadth-first search mimics the priority queue Q in Dijkstra.• Invariant: v comes after u in Q implies that
d[v] = d[u] or d[v] = d[u] + 1.
while Q ≠ ∅do u ← DEQUEUE(Q)
for each v ∈ Adj[u]do if d[v] = ∞
then d[v] ← d[u] + 1ENQUEUE(Q, v)
Introduction to Algorithms6.046J/18.401J
Prof. Charles E. Leiserson
LECTURE 15Shortest Paths II• Bellman-Ford algorithm• DAG shortest paths• Linear programming and
difference constraints• VLSI layout compaction
Introduction to Algorithms November 3, 2004 L15.2© 2001–4 by Charles E. Leiserson
Negative-weight cyclesRecall: If a graph G = (V, E) contains a negative-weight cycle, then some shortest paths may not exist.Example:
uu vv
…
< 0
Introduction to Algorithms November 3, 2004 L15.3© 2001–4 by Charles E. Leiserson
Negative-weight cyclesRecall: If a graph G = (V, E) contains a negative-weight cycle, then some shortest paths may not exist.Example:
uu vv
…
< 0
Bellman-Ford algorithm: Finds all shortest-path lengths from a source s ∈ V to all v ∈ V or determines that a negative-weight cycle exists.
Introduction to Algorithms November 3, 2004 L15.4© 2001–4 by Charles E. Leiserson
Bellman-Ford algorithmd[s] ← 0for each v ∈ V – s
do d[v] ←∞
for i ← 1 to |V| – 1do for each edge (u, v) ∈ E
do if d[v] > d[u] + w(u, v)then d[v] ← d[u] + w(u, v)
for each edge (u, v) ∈ Edo if d[v] > d[u] + w(u, v)
then report that a negative-weight cycle exists
initialization
At the end, d[v] = δ(s, v), if no negative-weight cycles. Time = O(VE).
relaxation step
Introduction to Algorithms November 3, 2004 L15.5© 2001–4 by Charles E. Leiserson
Example of Bellman-Ford
AA
BB
EE
CC DD
–1
4
12
–3
2
5
3
Introduction to Algorithms November 3, 2004 L15.6© 2001–4 by Charles E. Leiserson
Example of Bellman-Ford
AA
BB
EE
CC DD
–1
4
12
–3
2
5
3
∞
0 ∞
∞ ∞
Initialization.
Introduction to Algorithms November 3, 2004 L15.7© 2001–4 by Charles E. Leiserson
Example of Bellman-Ford
AA
BB
EE
CC DD
–1
4
12
–3
2
5
3
∞
0 ∞
∞ ∞
1
2
34
5
7
8
Order of edge relaxation.
6
Introduction to Algorithms November 3, 2004 L15.8© 2001–4 by Charles E. Leiserson
Example of Bellman-Ford
AA
BB
EE
CC DD
–1
4
12
–3
2
5
3
∞
0 ∞
∞ ∞
1
2
34
5
7
8
6
Introduction to Algorithms November 3, 2004 L15.9© 2001–4 by Charles E. Leiserson
Example of Bellman-Ford
AA
BB
EE
CC DD
–1
4
12
–3
2
5
3
∞
0 ∞
∞ ∞
1
2
34
5
7
8
6
Introduction to Algorithms November 3, 2004 L15.10© 2001–4 by Charles E. Leiserson
Example of Bellman-Ford
AA
BB
EE
CC DD
–1
4
12
–3
2
5
3
∞
0 ∞
∞ ∞
1
2
34
5
7
8
6
Introduction to Algorithms November 3, 2004 L15.11© 2001–4 by Charles E. Leiserson
∞−1
Example of Bellman-Ford
AA
BB
EE
CC DD
–1
4
12
–3
2
5
30 ∞
∞ ∞
1
2
34
5
7
8
6
Introduction to Algorithms November 3, 2004 L15.12© 2001–4 by Charles E. Leiserson
∞4
−1
Example of Bellman-Ford
AA
BB
EE
CC DD
–1
4
12
–3
2
5
30 ∞
∞
1
2
34
5
7
8
6
Introduction to Algorithms November 3, 2004 L15.13© 2001–4 by Charles E. Leiserson
4
−1
Example of Bellman-Ford
AA
BB
EE
CC DD
–1
4
12
–3
2
5
30 ∞
∞
1
2
34
5
7
8
6
Introduction to Algorithms November 3, 2004 L15.14© 2001–4 by Charles E. Leiserson
42
−1
Example of Bellman-Ford
AA
BB
EE
CC DD
–1
4
12
–3
2
5
30 ∞
∞
1
2
34
5
7
8
6
Introduction to Algorithms November 3, 2004 L15.15© 2001–4 by Charles E. Leiserson
2
−1
Example of Bellman-Ford
AA
BB
EE
CC DD
–1
4
12
–3
2
5
30 ∞
∞
1
2
34
5
7
8
6
Introduction to Algorithms November 3, 2004 L15.16© 2001–4 by Charles E. Leiserson
2
−1
Example of Bellman-Ford
AA
BB
EE
CC DD
–1
4
12
–3
2
5
30 ∞
∞
1
2
34
5
7
8
End of pass 1.
6
Introduction to Algorithms November 3, 2004 L15.17© 2001–4 by Charles E. Leiserson
∞1
2
−1
Example of Bellman-Ford
AA
BB
EE
CC DD
–1
4
12
–3
2
5
30
∞
1
2
34
5
7
8
6
Introduction to Algorithms November 3, 2004 L15.18© 2001–4 by Charles E. Leiserson
1
2
−1
Example of Bellman-Ford
AA
BB
EE
CC DD
–1
4
12
–3
2
5
30
∞
1
2
34
5
7
8
6
Introduction to Algorithms November 3, 2004 L15.19© 2001–4 by Charles E. Leiserson
∞1
1
2
−1
Example of Bellman-Ford
AA
BB
EE
CC DD
–1
4
12
–3
2
5
30
1
2
34
5
7
8
6
Introduction to Algorithms November 3, 2004 L15.20© 2001–4 by Charles E. Leiserson
1
1
2
−1
Example of Bellman-Ford
AA
BB
EE
CC DD
–1
4
12
–3
2
5
30
1
2
34
5
7
8
6
Introduction to Algorithms November 3, 2004 L15.21© 2001–4 by Charles E. Leiserson
1
1
2
−1
Example of Bellman-Ford
AA
BB
EE
CC DD
–1
4
12
–3
2
5
30
1
2
34
5
7
8
6
Introduction to Algorithms November 3, 2004 L15.22© 2001–4 by Charles E. Leiserson
1
1
2
−1
Example of Bellman-Ford
AA
BB
EE
CC DD
–1
4
12
–3
2
5
30
1
2
34
5
7
8
6
Introduction to Algorithms November 3, 2004 L15.23© 2001–4 by Charles E. Leiserson
1
1
2
−1
Example of Bellman-Ford
AA
BB
EE
CC DD
–1
4
12
–3
2
5
30
1
2
34
5
7
8
6
Introduction to Algorithms November 3, 2004 L15.24© 2001–4 by Charles E. Leiserson
1−2
1
2
−1
Example of Bellman-Ford
AA
BB
EE
CC DD
–1
4
12
–3
2
5
30
1
2
34
5
7
8
6
Introduction to Algorithms November 3, 2004 L15.25© 2001–4 by Charles E. Leiserson
−2
1
2
−1
Example of Bellman-Ford
AA
BB
EE
CC DD
–1
4
12
–3
2
5
30
1
2
34
5
7
8
6
End of pass 2 (and 3 and 4).
Introduction to Algorithms November 3, 2004 L15.26© 2001–4 by Charles E. Leiserson
CorrectnessTheorem. If G = (V, E) contains no negative-weight cycles, then after the Bellman-Ford algorithm executes, d[v] = δ(s, v) for all v ∈ V.
Introduction to Algorithms November 3, 2004 L15.27© 2001–4 by Charles E. Leiserson
CorrectnessTheorem. If G = (V, E) contains no negative-weight cycles, then after the Bellman-Ford algorithm executes, d[v] = δ(s, v) for all v ∈ V. Proof. Let v ∈ V be any vertex, and consider a shortest path p from s to v with the minimum number of edges.
v1v1
v2v2
v3v3 vk
vkv0v0
…s
v
p:
Since p is a shortest path, we haveδ(s, vi) = δ(s, vi–1) + w(vi–1, vi) .
Introduction to Algorithms November 3, 2004 L15.28© 2001–4 by Charles E. Leiserson
Correctness (continued)
v1v1
v2v2
v3v3 vk
vkv0v0
…s
v
p:
Initially, d[v0] = 0 = δ(s, v0), and d[v0] is unchanged by subsequent relaxations (because of the lemma from Lecture 14 that d[v] ≥ δ(s, v)).• After 1 pass through E, we have d[v1] = δ(s, v1).• After 2 passes through E, we have d[v2] = δ(s, v2).M
• After k passes through E, we have d[vk] = δ(s, vk).Since G contains no negative-weight cycles, p is simple. Longest simple path has ≤ |V| – 1 edges.
Introduction to Algorithms November 3, 2004 L15.29© 2001–4 by Charles E. Leiserson
Detection of negative-weight cycles
Corollary. If a value d[v] fails to converge after |V| – 1 passes, there exists a negative-weight cycle in G reachable from s.
Introduction to Algorithms November 3, 2004 L15.30© 2001–4 by Charles E. Leiserson
Linear programming
Let A be an m×n matrix, b be an m-vector, and cbe an n-vector. Find an n-vector x that maximizes cTx subject to Ax ≤ b, or determine that no such solution exists.
. ≤ .maximizingm
n
A x ≤ b cT x
Introduction to Algorithms November 3, 2004 L15.31© 2001–4 by Charles E. Leiserson
Linear-programming algorithms
Algorithms for the general problem• Simplex methods — practical, but worst-case
exponential time.• Interior-point methods — polynomial time and
competes with simplex.
Introduction to Algorithms November 3, 2004 L15.32© 2001–4 by Charles E. Leiserson
Linear-programming algorithms
Algorithms for the general problem• Simplex methods — practical, but worst-case
exponential time.• Interior-point methods — polynomial time and
competes with simplex.
Feasibility problem: No optimization criterion. Just find x such that Ax ≤ b.• In general, just as hard as ordinary LP.
Introduction to Algorithms November 3, 2004 L15.33© 2001–4 by Charles E. Leiserson
Solving a system of difference constraints
Linear programming where each row of A contains exactly one 1, one –1, and the rest 0’s. Example:
x1 – x2 ≤ 3x2 – x3 ≤ –2x1 – x3 ≤ 2
xj – xi ≤ wij
Introduction to Algorithms November 3, 2004 L15.34© 2001–4 by Charles E. Leiserson
Solving a system of difference constraints
Linear programming where each row of A contains exactly one 1, one –1, and the rest 0’s. Example:
x1 – x2 ≤ 3x2 – x3 ≤ –2x1 – x3 ≤ 2
xj – xi ≤ wij
Solution:x1 = 3x2 = 0x3 = 2
Introduction to Algorithms November 3, 2004 L15.35© 2001–4 by Charles E. Leiserson
Solving a system of difference constraints
Linear programming where each row of A contains exactly one 1, one –1, and the rest 0’s. Example:
x1 – x2 ≤ 3x2 – x3 ≤ –2x1 – x3 ≤ 2
xj – xi ≤ wij
Solution:x1 = 3x2 = 0x3 = 2
Constraint graph:
vjvjvi
vixj – xi ≤ wijwij
(The “A”matrix has dimensions|E | × |V |.)
Introduction to Algorithms November 3, 2004 L15.36© 2001–4 by Charles E. Leiserson
Unsatisfiable constraintsTheorem. If the constraint graph contains a negative-weight cycle, then the system of differences is unsatisfiable.
Introduction to Algorithms November 3, 2004 L15.37© 2001–4 by Charles E. Leiserson
Unsatisfiable constraintsTheorem. If the constraint graph contains a negative-weight cycle, then the system of differences is unsatisfiable.Proof. Suppose that the negative-weight cycle is v1 → v2 →L→ vk → v1. Then, we have
x2 – x1 ≤ w12x3 – x2 ≤ w23
Mxk – xk–1 ≤ wk–1, kx1 – xk ≤ wk1
Introduction to Algorithms November 3, 2004 L15.38© 2001–4 by Charles E. Leiserson
Unsatisfiable constraintsTheorem. If the constraint graph contains a negative-weight cycle, then the system of differences is unsatisfiable.Proof. Suppose that the negative-weight cycle is v1 → v2 →L→ vk → v1. Then, we have
x2 – x1 ≤ w12x3 – x2 ≤ w23
Mxk – xk–1 ≤ wk–1, kx1 – xk ≤ wk1
Therefore, no values for the xican satisfy the constraints.
0 ≤ weight of cycle< 0
Introduction to Algorithms November 3, 2004 L15.39© 2001–4 by Charles E. Leiserson
Satisfying the constraintsTheorem. Suppose no negative-weight cycle exists in the constraint graph. Then, the constraints are satisfiable.
Introduction to Algorithms November 3, 2004 L15.40© 2001–4 by Charles E. Leiserson
Satisfying the constraintsTheorem. Suppose no negative-weight cycle exists in the constraint graph. Then, the constraints are satisfiable.Proof. Add a new vertex s to V with a 0-weight edge to each vertex vi ∈ V.
v1v1
v4v4
v7v7
v9v9
v3v3
Introduction to Algorithms November 3, 2004 L15.41© 2001–4 by Charles E. Leiserson
Satisfying the constraintsTheorem. Suppose no negative-weight cycle exists in the constraint graph. Then, the constraints are satisfiable.Proof. Add a new vertex s to V with a 0-weight edge to each vertex vi ∈ V.
v1v1
v4v4
v7v7
v9v9
v3v3
s
0 Note:No negative-weight cycles introduced ⇒shortest paths exist.
Introduction to Algorithms November 3, 2004 L15.42© 2001–4 by Charles E. Leiserson
The triangle inequality gives us δ(s,vj) ≤ δ(s, vi) + wij. Since xi = δ(s, vi) and xj = δ(s, vj), the constraint xj – xi≤ wij is satisfied.
Proof (continued)Claim: The assignment xi = δ(s, vi) solves the constraints.
ss
vjvj
vivi
δ(s, vi)
δ(s, vj) wij
Consider any constraint xj – xi ≤ wij, and consider the shortest paths from s to vj and vi:
Introduction to Algorithms November 3, 2004 L15.43© 2001–4 by Charles E. Leiserson
Bellman-Ford and linear programming
Corollary. The Bellman-Ford algorithm can solve a system of m difference constraints on nvariables in O(mn) time. Single-source shortest paths is a simple LP problem.In fact, Bellman-Ford maximizes x1 + x2 + L + xnsubject to the constraints xj – xi ≤ wij and xi ≤ 0(exercise).Bellman-Ford also minimizes maxixi – minixi(exercise).
Introduction to Algorithms November 3, 2004 L15.44© 2001–4 by Charles E. Leiserson
Application to VLSI layout compaction
Integrated-circuit features:
Problem: Compact (in one dimension) the space between the features of a VLSI layout without bringing any features too close together.
minimum separation λ
Introduction to Algorithms November 3, 2004 L15.45© 2001–4 by Charles E. Leiserson
VLSI layout compaction
11
x1 x2
2
d1
Constraint: x2 – x1 ≥ d1 + λBellman-Ford minimizes maxixi – minixi, which compacts the layout in the x-dimension.
Introduction to Algorithms6.046J/18.401J
Prof. Charles E. Leiserson
LECTURE 16Shortest Paths III• All-pairs shortest paths• Matrix-multiplication
algorithm• Floyd-Warshall algorithm• Johnson’s algorithm
Introduction to Algorithms November 8, 2004 L16.2© 2001–4 by Charles E. Leiserson
Shortest pathsSingle-source shortest paths• Nonnegative edge weights
Dijkstra’s algorithm: O(E + V lg V)• General
Bellman-Ford algorithm: O(VE)• DAG
One pass of Bellman-Ford: O(V + E)
Introduction to Algorithms November 8, 2004 L16.3© 2001–4 by Charles E. Leiserson
Shortest pathsSingle-source shortest paths• Nonnegative edge weights
Dijkstra’s algorithm: O(E + V lg V)• General
Bellman-Ford: O(VE)• DAG
One pass of Bellman-Ford: O(V + E)All-pairs shortest paths• Nonnegative edge weights
Dijkstra’s algorithm |V| times: O(VE + V 2 lg V)• General
Three algorithms today.
Introduction to Algorithms November 8, 2004 L16.4© 2001–4 by Charles E. Leiserson
All-pairs shortest paths
Input: Digraph G = (V, E), where V = 1, 2, …, n, with edge-weight function w : E → R.Output: n × n matrix of shortest-path lengths δ(i, j) for all i, j ∈ V.
Introduction to Algorithms November 8, 2004 L16.5© 2001–4 by Charles E. Leiserson
All-pairs shortest paths
Input: Digraph G = (V, E), where V = 1, 2, …, n, with edge-weight function w : E → R.Output: n × n matrix of shortest-path lengths δ(i, j) for all i, j ∈ V.IDEA:• Run Bellman-Ford once from each vertex.• Time = O(V 2E).• Dense graph (n2 edges) ⇒Θ(n 4) time in the
worst case.Good first try!
Introduction to Algorithms November 8, 2004 L16.6© 2001–4 by Charles E. Leiserson
Dynamic programmingConsider the n × n adjacency matrix A = (aij)of the digraph, and define
dij(0) = 0 if i = j,
∞ if i ≠ j;
Claim: We have
and for m = 1, 2, …, n – 1,dij
(m) = minkdik(m–1) + akj .
dij(m) = weight of a shortest path from
i to j that uses at most m edges.
Introduction to Algorithms November 8, 2004 L16.7© 2001–4 by Charles E. Leiserson
Proof of claimdij
(m) = minkdik(m–1) + akj
ii jjiM
k’s
≤ m – 1 edges
≤ m – 1 edges
≤ m – 1 edges
≤ m – 1 edges
Introduction to Algorithms November 8, 2004 L16.8© 2001–4 by Charles E. Leiserson
Proof of claimdij
(m) = minkdik(m–1) + akj
ii jjiM
k’s
≤ m – 1 edges
≤ m – 1 edges
≤ m – 1 edges
≤ m – 1 edges
Relaxation!for k ← 1 to n
do if dij > dik + akjthen dij ← dik + akj
Introduction to Algorithms November 8, 2004 L16.9© 2001–4 by Charles E. Leiserson
Proof of claimdij
(m) = minkdik(m–1) + akj
ii jjiM
k’s
≤ m – 1 edges
≤ m – 1 edges
≤ m – 1 edges
≤ m – 1 edges
Relaxation!for k ← 1 to n
do if dij > dik + akjthen dij ← dik + akj
Note: No negative-weight cycles impliesδ(i, j) = dij
(n–1) = dij (n) = dij
(n+1) = L
Introduction to Algorithms November 8, 2004 L16.10© 2001–4 by Charles E. Leiserson
Matrix multiplicationCompute C = A · B, where C, A, and B are n × nmatrices:
∑=
=n
kkjikij bac
1.
Time = Θ(n3) using the standard algorithm.
Introduction to Algorithms November 8, 2004 L16.11© 2001–4 by Charles E. Leiserson
Matrix multiplicationCompute C = A · B, where C, A, and B are n × nmatrices:
∑=
=n
kkjikij bac
1.
Time = Θ(n3) using the standard algorithm.What if we map “+” → “min” and “·” → “+”?
Introduction to Algorithms November 8, 2004 L16.12© 2001–4 by Charles E. Leiserson
Matrix multiplicationCompute C = A · B, where C, A, and B are n × nmatrices:
∑=
=n
kkjikij bac
1.
Time = Θ(n3) using the standard algorithm.What if we map “+” → “min” and “·” → “+”?
cij = mink aik + bkj.Thus, D(m) = D(m–1) “×” A.
Identity matrix = I =
∞∞∞∞∞∞∞∞∞∞∞∞
00
00
= D0 = (dij(0)).
Introduction to Algorithms November 8, 2004 L16.13© 2001–4 by Charles E. Leiserson
Matrix multiplication (continued)
The (min, +) multiplication is associative, and with the real numbers, it forms an algebraic structure called a closed semiring.Consequently, we can compute
D(1) = D(0) · A = A1
D(2) = D(1) · A = A2
M MD(n–1) = D(n–2) · A = An–1 ,
yielding D(n–1) = (δ(i, j)).Time = Θ(n·n3) = Θ(n4). No better than n × B-F.
Introduction to Algorithms November 8, 2004 L16.14© 2001–4 by Charles E. Leiserson
Improved matrix multiplication algorithm
Repeated squaring: A2k = Ak × Ak.Compute A2, A4, …, A2lg(n–1) .
O(lg n) squarings
Time = Θ(n3 lg n).
To detect negative-weight cycles, check the diagonal for negative values in O(n) additional time.
Note: An–1 = An = An+1 = L.
Introduction to Algorithms November 8, 2004 L16.15© 2001–4 by Charles E. Leiserson
Floyd-Warshall algorithm
Also dynamic programming, but faster!
Define cij(k) = weight of a shortest path from i
to j with intermediate vertices belonging to the set 1, 2, …, k.
ii ≤ k≤ k ≤ k≤ k ≤ k≤ k ≤ k≤ k jj
Thus, δ(i, j) = cij(n). Also, cij
(0) = aij .
Introduction to Algorithms November 8, 2004 L16.16© 2001–4 by Charles E. Leiserson
Floyd-Warshall recurrencecij
(k) = min cij(k–1), cik
(k–1) + ckj(k–1)
ii jj
k
icij
(k–1)
cik(k–1) ckj
(k–1)
intermediate vertices in 1, 2, …, k
Introduction to Algorithms November 8, 2004 L16.17© 2001–4 by Charles E. Leiserson
Pseudocode for Floyd-Warshall
for k ← 1 to ndo for i ← 1 to n
do for j ← 1 to ndo if cij > cik + ckj
then cij ← cik + ckjrelaxation
Notes:• Okay to omit superscripts, since extra relaxations
can’t hurt.• Runs in Θ(n3) time.• Simple to code.• Efficient in practice.
Introduction to Algorithms November 8, 2004 L16.18© 2001–4 by Charles E. Leiserson
Transitive closure of a directed graph
Compute tij = 1 if there exists a path from i to j,0 otherwise.
IDEA: Use Floyd-Warshall, but with (∨, ∧) instead of (min, +):
tij(k) = tij(k–1) ∨ (tik(k–1) ∧ tkj(k–1)).
Time = Θ(n3).
Introduction to Algorithms November 8, 2004 L16.19© 2001–4 by Charles E. Leiserson
Graph reweightingTheorem. Given a function h : V → R, reweight each edge (u, v) ∈ E by wh(u, v) = w(u, v) + h(u) – h(v). Then, for any two vertices, all paths between them are reweighted by the same amount.
Introduction to Algorithms November 8, 2004 L16.20© 2001–4 by Charles E. Leiserson
Graph reweighting
Proof. Let p = v1 → v2 →L→ vk be a path in G. We have
( )
)()()(
)()(),(
)()(),(
),()(
1
1
1
11
1
111
1
11
k
k
k
iii
k
iiiii
k
iiihh
vhvhpw
vhvhvvw
vhvhvvw
vvwpw
−+=
−+=
−+=
=
∑
∑
∑
−
=+
−
=++
−
=+
.
Theorem. Given a function h : V → R, reweight each edge (u, v) ∈ E by wh(u, v) = w(u, v) + h(u) – h(v). Then, for any two vertices, all paths between them are reweighted by the same amount.
Sameamount!
Introduction to Algorithms November 8, 2004 L16.21© 2001–4 by Charles E. Leiserson
Shortest paths in reweightedgraphs
Corollary. δh(u, v) = δ(u, v) + h(u) – h(v).
Introduction to Algorithms November 8, 2004 L16.22© 2001–4 by Charles E. Leiserson
Shortest paths in reweightedgraphs
Corollary. δh(u, v) = δ(u, v) + h(u) – h(v).
IDEA: Find a function h : V → R such that wh(u, v) ≥ 0 for all (u, v) ∈ E. Then, run Dijkstra’s algorithm from each vertex on the reweighted graph. NOTE: wh(u, v) ≥ 0 iff h(v) – h(u) ≤ w(u, v).
Introduction to Algorithms November 8, 2004 L16.23© 2001–4 by Charles E. Leiserson
Johnson’s algorithm1. Find a function h : V → R such that wh(u, v) ≥ 0 for
all (u, v) ∈ E by using Bellman-Ford to solve the difference constraints h(v) – h(u) ≤ w(u, v), or determine that a negative-weight cycle exists.• Time = O(VE).
2. Run Dijkstra’s algorithm using wh from each vertex u ∈ V to compute δh(u, v) for all v ∈ V.• Time = O(VE + V 2 lg V).
3. For each (u, v) ∈ V × V, computeδ(u, v) = δh(u, v) – h(u) + h(v) .
• Time = O(V 2).Total time = O(VE + V 2 lg V).
Introduction to Algorithms6.046J/18.401
Lecture 18Prof. Piotr Indyk
Today
• We have seen algorithms for:– “numerical” data (sorting,
median) – graphs (shortest path, MST)
• Today and the next lecture: algorithms for geometric data
© 2003 by Piotr Indyk Introduction to Algorithms November 10, 2004 L17.2
Computational Geometry
• Algorithms for geometric problems• Applications: CAD, GIS, computer
vision,……. • E.g., the closest pair problem:
– Given: a set of points P=p1…pn in the plane, such that pi=(xi,yi)
– Goal: find a pair pi ≠pj that minimizes ||pi – pj|| ||p-q||= [(px-qx)2+(py-qy)2]1/2
• We will see more examples in the nextlecture
© 2003 by Piotr Indyk Introduction to Algorithms November 10, 2004 L17.3
Closest Pair
• Find a closest pair among p1…pn
• Easy to do in O(n2) time – For all pi ≠pj, compute ||pi – pj|| and
choose the minimum • We will aim for O(n log n) time
© 2003 by Piotr Indyk Introduction to Algorithms November 10, 2004 L17.4
Divide and conquer
• Divide: – Compute the median of
x-coordinates – Split the points into PL
and PR, each of size n/2 • Conquer: compute the
closest pairs for PL and PR
• Combine the results (the hard part)
© 2003 by Piotr Indyk Introduction to Algorithms November 10, 2004 L17.5
Combine2d
• Let d=min(d1,d2) • Observe:
– Need to check only pairs which cross the dividingline
– Only interested in pairs within distance < d
• Suffices to look at points in the2d-width strip around themedian line
d1
d2
© 2003 by Piotr Indyk Introduction to Algorithms November 10, 2004 L17.6
Scanning the strip
• Sort all points in the strip by their y-coordinates, forming q1…qk, k ≤ n.
dd
d• Let yi be the y-coordinate of qi d• dmin= d
d
• For i=1 to k– j=i-1 – While yi-yj < d
• If ||qi–qj||<d then dmin=||qi–qj|| • j:=j-1
• Report dmin (and the corresponding pair)
© 2003 by Piotr Indyk Introduction to Algorithms November 10, 2004 L17.7
Analysis
• Correctness: easy• Running time is more
involved • Can we have many qj’s
that are within distance d from qi ?
• No • Proof by packing
argument
d
© 2003 by Piotr Indyk Introduction to Algorithms November 10, 2004 L17.8
Analysis, ctd.
Theorem: there are at most 7 qj’s such that yi-yj ≤ d.
Proof:• Each such qj must lie either in
the left or in the right d × d square
• Within each square, all pointshave distance distance ≥ d from others
• We can pack at most 4 such points into one square, so wehave 8 points total (incl. qi)
qi
© 2003 by Piotr Indyk Introduction to Algorithms November 10, 2004 L17.9
Packing bound
• Proving “4” is not easy• Will prove “5”
– Draw a disk of radius d/2around each point
– Disks are disjoint– The disk-square intersection
has area ≥ π (d/2)2/4 = π/16 d2
– The square has area d2
– Can pack at most 16/π ≈ 5.1 points
© 2003 by Piotr Indyk Introduction to Algorithms November 10, 2004 L17.10
Running time
• Divide: O(n) • Combine: O(n log n) because we sort by y• However, we can:
– Sort all points by y at the beginning– Divide preserves the y-order of points Then combine takes only O(n)
• We get T(n)=2T(n/2)+O(n), soT(n)=O(n log n)
© 2003 by Piotr Indyk Introduction to Algorithms November 10, 2004 L17.11
Close pair
• Given: P=p1…pn • Goal: check if there is any pair pi ≠pj within
distance R from each other • Will give an O(n) time algorithm, using…
…radix sort ! (assuming coordinates are small integers)
© 2003 by Piotr Indyk Introduction to Algorithms November 10, 2004 L17.12
Algorithm
• Impose a square grid onto the plane, where each cell is an R × R square
• Put each point into a bucket corresponding to the cell itbelongs to. That is:
– For each point p=(x,y), create computes its bucketID b(p)=( x/R , y/R )
– Radix sort all b(p) ’s – Each sequence of the same b(p) forms a bucket
• If there is a bucket with > 4 points in it, answer YESand exit
• Otherwise, for each p∈P:– Let c =b(p)– Let C be the set of bucket IDs of the 8 cells
adjacent to c – For all points q from buckets in C ∪c
• If ||p-q||≤R, then answer YES and exit • Answer NO
(1,1), (1,2), (1,2), (2,1), (2,2), (2,2), (2,3), (3,1), (3,2) © 2003 by Piotr Indyk Introduction to Algorithms November 10, 2004 L17.13
Bucket access
• Given a bucket ID c, how can we quickly retrieve all points p such that b(p)=c ?
• This is exactly the dictionary problem (Lecture 7)
• E.g., we can use hashing.
© 2003 by Piotr Indyk Introduction to Algorithms November 10, 2004 L17.14
Analysis
• Running time: – Putting points into the buckets: O(n) time – Checking if there is a heavy bucket: O(n) – Checking the cells: 9 × 4 × n = O(n)
• Overall: linear time
© 2003 by Piotr Indyk Introduction to Algorithms November 10, 2004 L17.15
Computational Model
• In the two lectures, we assume that – The input (e.g., point coordinates) are real numbers – We can perform (natural) operations on them in
constant time, with perfect precision • Advantage: simplicity • Drawbacks: highly non-trivial issues:
– Theoretical: if we allow arbitrary operations on reals, we can compress n numbers into a one number
– Practical: algorithm designed for infinite precision sometimes fail on real computers
© 2003 by Piotr Indyk Introduction to Algorithms November 10, 2004 L17.16
Introduction to Algorithms6.046J/18.401
Lecture 17Prof. Piotr Indyk
Computational Geometry ctd.
• Segment intersection problem:– Given: a set of n distinct
segments s1…sn, represented by coordinates of endpoints
– Detection: detect if there is any pair si ≠ sj that intersects
– Reporting: report all pairs of intersecting segments
© 2003 by Piotr Indyk Introduction to Algorithms November 15, 2004 L18.2
Segment intersection
• Easy to solve in O(n2) time • Is it possible to get a better algorithm
for the reporting problem ? • NO (in the worst-case)• However:
– We will see we can do better for the detection problem
– Moreover, the number of intersections P is usually small. Then, we would like an outputsensitive algorithm, whoserunning time is low if P is small.
© 2003 by Piotr Indyk Introduction to Algorithms November 15, 2004 L18.3
Result
• We will show: – O(n log n) time for detection – O( (n +P) log n) time for reporting
• We will use …
… (no, not divide and conquer)… Binary Search Trees
• Specifically: Line sweep approach
© 2003 by Piotr Indyk Introduction to Algorithms November 15, 2004 L18.4
Orthogonal segmentsV-segment
• All segments are either horizontal or vertical
H-segment • Assumption: all coordinates are distinct
• Therefore, only vertical-horizontal intersections exist
© 2003 by Piotr Indyk Introduction to Algorithms November 15, 2004 L18.5
Orthogonal segments• Sweep line:
– A vertical line sweeps theplane from left to right
– It “stops” at all “important” x-coordinates, i.e., when ithits a V-segment orendpoints of an H-segment
– Invariant: all intersections on the left side of the sweep linehave been already reported
© 2003 by Piotr Indyk Introduction to Algorithms November 15, 2004 L18.6
Orthogonal segments ctd.
• We maintain sorted y-coordinates of H-segmentscurrently intersected by thesweep line (using a balanced 17BST V)
• When we hit the left point of 12
an H-segment, we add its y-coordinate to V
• When we hit the right point ofan H-segment, we delete its y-coordinate from V
© 2003 by Piotr Indyk Introduction to Algorithms November 15, 2004 L18.7
Orthogonal segments ctd.ytop
y
• Whenever we hit a V-segment having coord.
top, ybot), we report all 17H-segments in V with y-
coordinates in [ytop, ybot] ybot
12
© 2003 by Piotr Indyk Introduction to Algorithms November 15, 2004 L18.8
Algorithm
• Sort all V-segments and endpoints of H-segments by their x-coordinates – this gives the “trajectory” of the sweep line
• Scan the elements in the sorted list: – Left endpoint: add segment to tree V – Right endpoint: remove segment from V – V-segment: report intersections with the
H-segments stored in V
© 2003 by Piotr Indyk Introduction to Algorithms November 15, 2004 L18.9
Analysis
• Sorting: O(n log n)• Add/delete H-segments to/from vertical data
structure V: – O(log n) per operation– O(n log n) total
• Processing V-segments: – O(log n) per intersection - SEE NEXT SLIDE – O(P log n) total
• Overall: O( (P+ n) log n) time • Can be improved to O(P +n log n)
© 2003 by Piotr Indyk Introduction to Algorithms November 15, 2004 L18.10
Analyzing intersections
• Given: – A BST V containing y-coordinates – An interval I=[ybot,ytop]
• Goal: report all y’s in V that belong to I • Algorithm:
– y=Successor(ybot) – While y≤ytop
• Report y • y:=Successor(y)
– End • Time: (number of reported y’s)*O(log n) + O(log n)
© 2003 by Piotr Indyk Introduction to Algorithms November 15, 2004 L18.11
The general case
• Assumption: all coordinates of endpoints and intersections distinct
• In particular: – No vertical segments – No three segments
intersect at one point
© 2003 by Piotr Indyk Introduction to Algorithms November 15, 2004 L18.12
Sweep line
• Invariant (as before): all intersectionson the left of the sweep line havebeen already reported
• Stops at all “important” x-coordinates, i.e., when it hitsendpoints or intersections
• Do not know the intersections in advance !
• The list of intersection coordinates is constructed and maintained dynamically (in a “horizontal” data structure H)
© 2003 by Piotr Indyk Introduction to Algorithms November 15, 2004 L18.13
Sweep line
• Also need to maintain the information about the segmentsintersecting the sweep line
• Cannot keep the values of y-coordinates of the segments !
• Instead, we will maintain theirorder .I.e., at any point, wemaintain all segments intersectingthe sweep line, sorted by the y-coordinates of the intersections
(in a “vertical” data structure V)
© 2003 by Piotr Indyk Introduction to Algorithms November 15, 2004 L18.14
Algorithm
• Initialize the “vertical” BST V (to “empty”)• Initialize the “horizontal” priority queue H (to contain the
segments’ endpoints sorted by x-coordinates) • Repeat
– Take the next “event” p from H:// Update V– If p is the left endpoint of a segment, add the segment
to V – If p is the right endpoint of a segment, remove the
segment from V – If p is the intersection point of s and s’, swap the order
of s and s’ in V, report p
© 2003 by Piotr Indyk Introduction to Algorithms November 15, 2004 L18.15
Algorithm ctd.
// Update H – For each new pair of neighbors s and s’ in V:
• Check if s and s’ intersect on the right side of the sweep line
• If so, add their intersection point to H • Remove the possible duplicates in H
• Until H is empty
© 2003 by Piotr Indyk Introduction to Algorithms November 15, 2004 L18.16
Analysis
• Initializing H: O(n log n)• Updating V:
– O(log n) per operation – O( (P+n) log n) total
• Updating H: – O(log n) per intersection – O(P log n) total
• Overall: O( (P+ n) log n) time
© 2003 by Piotr Indyk Introduction to Algorithms November 15, 2004 L18.17
Correctness
• All reported intersections are correct• Assume there is an intersection not reported. Let p=(x,y)
be the first such unreported intersection (of s and s’ )• Let x’ be the last event before p. Observe that:
– At time x’ segments s and s’ are neighbors on thesweep line
– Since no intersections were missed till then, V maintained the right order of intersecting segments
– Thus, s and s’ were neighbors in V at time x’. Thus,their intersection should have been detected
© 2003 by Piotr Indyk Introduction to Algorithms November 15, 2004 L18.18
Changes
• Y’s – change the order
© 2003 by Piotr Indyk Introduction to Algorithms November 15, 2004 L18.19
Introduction to Algorithms6.046J/18.401J
LECTURE 19Take-home exam • Instructions • Academic honesty • Strategies for doing well
Prof. Charles E. Leiserson
Take-home quiz
The take-home quiz contains 5 problems worth25 points each, for a total of 125 points.
• 1 easy • 2 moderate• 1 hard • 1 very hard
© 2001–4 by Charles E. Leiserson Introduction to Algorithms November 17, 2004 L19.2
End of quiz
• Your exam is due between 10:00 and 11:00 A.M. on Monday, November 22,2004.
• Late exams will not be accepted unlessyou obtain a Dean’s Excuse or makeprior arrangements with yourrecitation instructor.
• You must hand in your own exam in person.
© 2001–4 by Charles E. Leiserson Introduction to Algorithms November 17, 2004 L19.3
Planning
• The quiz should take you about 12 hours todo, but you have five days in which to do it.
• Plan your time wisely. Do not overwork,and get enough sleep.
• Ample partial credit will be given for goodsolutions, especially if they are well written.
• The better your asymptotic running-timebounds, the higher your score.
• Bonus points will be given for exceptionallyefficient or elegant solutions.
© 2001–4 by Charles E. Leiserson Introduction to Algorithms November 17, 2004 L19.4
Format• Each problem should be answered on
a separate sheet (or sheets) of 3-holepunched paper.
• Mark the top of each problem with • your name, • 6.046J/18.410J, • the problem number, • your recitation time, • and your TA.
© 2001–4 by Charles E. Leiserson Introduction to Algorithms November 17, 2004 L19.5
Executive summary
• Your solution to a problem should start witha topic paragraph that provides an executivesummary of your solution.
• This executive summary should describe • the problem you are solving, • the techniques you use to solve it, • any important assumptions you make, and • the running time your algorithm achieves.
© 2001–4 by Charles E. Leiserson Introduction to Algorithms November 17, 2004 L19.6
Solutions• Write up your solutions cleanly and concisely
to maximize the chance that we understand them.
• Be explicit about running time and algorithms. • For example, don't just say you sort n numbers,
state that you are using heapsort, which sorts the n numbers in O(n lg n) time in the worst case.
• When describing an algorithm, give an Englishdescription of the main idea of the algorithm.
• Use pseudocode only if necessary to clarify your solution.
© 2001–4 by Charles E. Leiserson Introduction to Algorithms November 17, 2004 L19.7
Solutions
• Give examples, and draw figures.• Provide succinct and convincing arguments
for the correctness of your solutions. • Do not regurgitate material presented in class. • Cite algorithms and theorems from CLRS,
lecture, and recitation to simplify yoursolutions.
© 2001–4 by Charles E. Leiserson Introduction to Algorithms November 17, 2004 L19.8
Assumptions
• Part of the goal of this exam is to testengineering common sense.
• If you find that a question is unclear orambiguous, make reasonable assumptionsin order to solve the problem.
• State clearly in your write-up whatassumptions you have made.
• Be careful what you assume, however,because you will receive little credit if youmake a strong assumption that renders aproblem trivial.
© 2001–4 by Charles E. Leiserson Introduction to Algorithms November 17, 2004 L19.9
Bugs, etc.• If you think that you’ve found a bug, please send
email. • Corrections and clarifications will be sent to the
class via email. • Check your email daily to avoid missing
potentially important announcements.• If you did not receive an email last night
reminding you about Quiz 2, then you are not onthe class email list. Please let your recitationinstructor know immediately.
© 2001–4 by Charles E. Leiserson Introduction to Algorithms November 17, 2004 L19.10
Academic honesty
• This quiz is “limited open book.”• You may use
• your course notes, • the CLRS textbook, • lecture videos, • basic reference materials such as dictionaries,
and • any of the handouts posted on the server.
• No other sources whatsoever may be consulted!
© 2001–4 by Charles E. Leiserson Introduction to Algorithms November 17, 2004 L19.11
Academic honesty
• For example, you may not use notes or solutionsfrom other times that this course or other related courses have been taught, or materials on the server.
• These materials will not help you, but you may notuse them anyhow.
• You may not communicate with any personexcept members of the 6.046 staff about anyaspect of the exam until after noon on Monday,November 22, even if you have already handedin your exam.
© 2001–4 by Charles E. Leiserson Introduction to Algorithms November 17, 2004 L19.12
Academic honesty
• If at any time you feel that you may haveviolated this policy, it is imperative that youcontact the course staff immediately.
• It will be much the worse for you if third partiesdivulge your indiscretion.
• If you have any questions about what resourcesmay or may not be used during the quiz, sendemail.
© 2001–4 by Charles E. Leiserson Introduction to Algorithms November 17, 2004 L19.13
Poll of 78 quiz takers
Question 1: “Did you cheat?”
• 76 — “No.” • 1 — “Yes.” • 1 — “Abstain.”
© 2001–4 by Charles E. Leiserson Introduction to Algorithms November 17, 2004 L19.14
Poll of 78 quiz takers
Question 2: “How many people do you knowwho cheated?”
• 72 — “None.” • 2 — “3 people compared answers.”• 1 — “Suspect 2, but don’t know.”• 1 — “Either 0 or 2.” • 1 — “Abstain.” • 1 — “10” (the cheater).
© 2001–4 by Charles E. Leiserson Introduction to Algorithms November 17, 2004 L19.15
Reread instructions
Please reread the exam instructions in their entirety at
least once a day during the exam.
© 2001–4 by Charles E. Leiserson Introduction to Algorithms November 17, 2004 L19.16
Test-taking strategies
• Manage your time. • Manage your psyche. • Brainstorm. • Write-up early and often.
© 2001–4 by Charles E. Leiserson Introduction to Algorithms November 17, 2004 L19.17
Manage your time
• Work on all problems the first day. • Budget time for write-ups and debugging.• Don’t get sucked into one problem at the
expense of others. • Replan your strategy every day.
© 2001–4 by Charles E. Leiserson Introduction to Algorithms November 17, 2004 L19.18
Manage your psyche
• Get enough sleep.• Maintain a patient, persistent, and
positive attitude. • Use adrenaline productively. • Relax, and have fun.
• It’s not the end of the world!
© 2001–4 by Charles E. Leiserson Introduction to Algorithms November 17, 2004 L19.19
Brainstorm• Get an upper bound, even if it is loose. • Look for analogies with problems you’ve seen.• Exploit special structure. • Solve a simpler problem. • Draw diagrams. • Contemplate. • Be wary of self-imposed constraints — think
“out of the box.” • Work out small examples, and abstract. • Understand things in two ways: sanity checks.
© 2001–4 by Charles E. Leiserson Introduction to Algorithms November 17, 2004 L19.20
Write up early and often
• Write up partial solutions. • Groom your work every day. • Work on shortening and simplifying.• Provide an executive summary. • Ample partial credit will be given! • Unnecessarily long answers will be
penalized.
© 2001–4 by Charles E. Leiserson Introduction to Algorithms November 17, 2004 L19.21
Positive attitude
© 2001–4 by Charles E. Leiserson Introduction to Algorithms November 17, 2004 L19.22
Introduction to Algorithms6.046J/18.401J
LECTURE 20Network Flow I• Flow networks• Maximum-flow problem • Flow notation• Properties of flow • Cuts • Residual networks • Augmenting paths
Prof. Charles E. Leiserson
Flow networksDefinition. A flow network is a directed graphG = (V, E) with two distinguished vertices: a source s and a sink t. Each edge (u, v) ∈ E has a nonnegative capacity c(u, v). If (u, v) ∉ E, then c(u, v) = 0.
© 2001–4 by Charles E. Leiserson Introduction to Algorithms November 24, 2004 L20.2
Flow networksDefinition. A flow network is a directed graphG = (V, E) with two distinguished vertices: a source s and a sink t. Each edge (u, v) ∈ E has a nonnegative capacity c(u, v). If (u, v) ∉ E, then c(u, v) = 0.
Example:
ss tt
3 2
3
3 2
2 3
31
2
1
© 2001–4 by Charles E. Leiserson Introduction to Algorithms November 24, 2004 L20.3
Flow networks
Definition. A positive flow on G is a function p : V × V → R satisfying the following: • Capacity constraint: For all u, v ∈ V,
0 ≤ p(u, v) ≤ c(u, v). • Flow conservation: For all u ∈ V – s, t,
∑ v u p ) − ∑ u v p ) = 0.( , ( ,v∈V v∈V
© 2001–4 by Charles E. Leiserson Introduction to Algorithms November 24, 2004 L20.4
Flow networks
Definition. A positive flow on G is a function p : V × V → R satisfying the following: • Capacity constraint: For all u, v ∈ V,
0 ≤ p(u, v) ≤ c(u, v). • Flow conservation: For all u ∈ V – s, t,
∑ v u p ) − ∑ u v p ) = 0.( , ( , v∈V v∈V
The value of a flow is the net flow out of the source:
∑ v s p ) − ∑ s v p ) .( , ( , v∈V v∈V
© 2001–4 by Charles E. Leiserson Introduction to Algorithms November 24, 2004 L20.5
A flow on a networkpositive capacityflow
1:1
2:2 1:3 2:3
ss 0:1 1:3 2:3 1:2 tt
2:2 1:22:3
© 2001–4 by Charles E. Leiserson Introduction to Algorithms November 24, 2004 L20.6
A flow on a networkpositive capacityflow
1:1
2:2 1:3 2:3
ss 0:1 1:3 2:3 1:2 tt
2:2 u 2:3 1:2
Flow conservation (like Kirchoff’s current law):• Flow into u is 2 + 1 = 3. • Flow out of u is 0 + 1 + 2 = 3. The value of this flow is 1 – 0 + 2 = 3.
© 2001–4 by Charles E. Leiserson Introduction to Algorithms November 24, 2004 L20.7
The maximum-flow problem
Maximum-flow problem: Given a flow network G, find a flow of maximum value on G.
2:2 2:3 2:3
ss 0:1 0:3 2:3 1:2 tt1:1
2:22:2 3:3
The value of the maximum flow is 4.© 2001–4 by Charles E. Leiserson Introduction to Algorithms November 24, 2004 L20.8
Flow cancellationWithout loss of generality, positive flow goes either from u to v, or from v to u, but not both.
vv
uu
2:3
vv
uu
1:31:2 0:2 Net flow from u to v in both cases is 1.
The capacity constraint and flow conservation are preserved by this transformation. INTUITION: View flow as a rate, not a quantity.
© 2001–4 by Charles E. Leiserson Introduction to Algorithms November 24, 2004 L20.9
A notational simplification
IDEA: Work with the net flow between two vertices, rather than with the positive flow.Definition. A (net) flow on G is a function f : V × V → R satisfying the following: • Capacity constraint: For all u, v ∈ V,
f (u, v) ≤ c(u, v). • Flow conservation: For all u ∈ V – s, t,
,∑ f ( v u ) = 0. v∈V
• Skew symmetry: For all u, v ∈ V, f (u, v) = –f (v, u).
© 2001–4 by Charles E. Leiserson Introduction to Algorithms November 24, 2004 L20.10
A notational simplification
IDEA: Work with the net flow between two vertices, rather than with the positive flow.Definition. A (net) flow on G is a function f : V × V → R satisfying the following: •
•
•
f (u, v) ≤ c(u, v).Flow conservation: For all u ∈ V – s, t,
, v∈V ∑ f ( v u ) = 0.
Skew symmetry: For all u, v ∈ V, f (u, v) = –f (v, u).
Capacity constraint: For all u, v ∈ V,
One summation instead of two.
© 2001–4 by Charles E. Leiserson Introduction to Algorithms November 24, 2004 L20.11
Equivalence of definitionsTheorem. The two definitions are equivalent.
© 2001–4 by Charles E. Leiserson Introduction to Algorithms November 24, 2004 L20.12
Equivalence of definitionsTheorem. The two definitions are equivalent. Proof. (⇒) Let f (u, v) = p(u, v) – p(v, u). • Capacity constraint: Since p(u, v) ≤ c(u, v) and
p(v, u) ≥ 0, we have f (u, v) ≤ c(u, v).• Flow conservation:
∑ f ( v u ) =∑( v u p ) − u v p )), ( , ( , v∈V v∈V
=∑ v u p ) −∑ u v p )( , ( , v∈V v∈V
• Skew symmetry:f (u, v) = p(u, v) – p(v, u)
= – (p(v, u) – p(u, v)) = – f (v, u).
© 2001–4 by Charles E. Leiserson Introduction to Algorithms November 24, 2004 L20.13
Proof (continued)(⇐) Let
f (u, v) if f(u, v) > 0,p(u, v) = 0 if f(u, v) ≤ 0.
• Capacity constraint: By definition, p(u, v) ≥ 0. Since f (u, v) ≤ c(u, v), it follows that p(u, v) ≤ c(u, v).
• Flow conservation: If f (u, v) > 0, then p(u, v) – p(v, u) = f (u, v). If f (u, v) ≤ 0, then p(u, v) – p(v, u) = – f (v, u) = f (u, v) by skew symmetry. Therefore,
∑ v u p ) −∑ u v p ) =∑ f ( v u ).( , ( , , v∈V v∈V v∈V
© 2001–4 by Charles E. Leiserson Introduction to Algorithms November 24, 2004 L20.14
Notation
Definition. The value of a flow f, denoted by | f |, is given by
f = ∑ f ( v s ), V v ∈
,= f ( V s ) .Implicit summation notation: A set used in an arithmetic formula represents a sum over the elements of the set. • Example — flow conservation:
f (u, V) = 0 for all u ∈ V – s, t.© 2001–4 by Charles E. Leiserson Introduction to Algorithms November 24, 2004 L20.15
Simple properties of flow
Lemma. • f (X, X) = 0, • f (X, Y) = – f (Y, X), • f (X∪Y, Z) = f (X, Z) + f (Y, Z) if X∩Y = ∅.
© 2001–4 by Charles E. Leiserson Introduction to Algorithms November 24, 2004 L20.16
Simple properties of flow
Lemma. • f (X, X) = 0, • f (X, Y) = – f (Y, X), • f (X∪Y, Z) = f (X, Z) + f (Y, Z) if X∩Y = ∅.
Theorem. | f | = f (V, t).Proof.
| f | = f (s, V) = f (V, V) – f (V–s, V) Omit braces. = f (V, V–s) = f (V, t) + f (V, V–s–t) = f (V, t).
© 2001–4 by Charles E. Leiserson Introduction to Algorithms November 24, 2004 L20.17
Flow into the sink
2:2 2:3 2:3
ss 0:1 0:3 1:3 0:2 tt1:1
2:22:2 3:3
| f | = f (s, V) = 4 f (V, t) = 4
© 2001–4 by Charles E. Leiserson Introduction to Algorithms November 24, 2004 L20.18
CutsDefinition. A cut (S, T) of a flow network G = (V, E) is a partition of V such that s ∈ S and t ∈ T. If f is a flow on G, then the flow across the cut is f (S, T).
2:2 2:3 2:3 ∈ S
ss 0:1 0:3 1:3 0:2 tt ∈ T1:1
2:22:2 3:3
f (S, T) = (2 + 2) + (– 2 + 1 – 1 + 2) = 4
© 2001–4 by Charles E. Leiserson Introduction to Algorithms November 24, 2004 L20.19
Another characterization of flow value
Lemma. For any flow f and any cut (S, T), we have | f | = f (S, T).
© 2001–4 by Charles E. Leiserson Introduction to Algorithms November 24, 2004 L20.20
Another characterization of flow value
Lemma. For any flow f and any cut (S, T), we have | f | = f (S, T). Proof. f (S, T) = f (S, V) – f (S, S)
= f (S, V) = f (s, V) + f (S–s, V) = f (s, V) = | f |.
© 2001–4 by Charles E. Leiserson Introduction to Algorithms November 24, 2004 L20.21
Capacity of a cutDefinition. The capacity of a cut (S, T) is c(S, T).
2:2 2:3 2:3 ∈ S
ss 0:1 0:3 1:3 0:2 tt ∈ T1:1
2:22:2 3:3
c(S, T) = (3 + 2) + (1 + 2 + 3) = 11
© 2001–4 by Charles E. Leiserson Introduction to Algorithms November 24, 2004 L20.22
Upper bound on the maximumflow value
Theorem. The value of any flow is bounded above by the capacity of any cut.
.
© 2001–4 by Charles E. Leiserson Introduction to Algorithms November 24, 2004 L20.23
Upper bound on the maximumflow value
Theorem. The value of any flow is bounded above by the capacity of any cut.
f = T S f )( ,Proof. ,= ∑∑ f ( v u )
S vu∈ ∈T ≤ ∑∑ v u c )( ,
S vu∈ ∈T ( ,= T S c ) .
© 2001–4 by Charles E. Leiserson Introduction to Algorithms November 24, 2004 L20.24
Residual networkDefinition. Let f be a flow on G = (V, E). The residual network Gf (V, Ef ) is the graph with strictly positive residual capacities
cf (u, v) = c(u, v) – f (u, v) > 0. Edges in Ef admit more flow.
© 2001–4 by Charles E. Leiserson Introduction to Algorithms November 24, 2004 L20.25
Residual networkDefinition. Let f be a flow on G = (V, E). The residual network Gf (V, Ef ) is the graph with strictly positive residual capacities
cf (u, v) = c(u, v) – f (u, v) > 0. Edges in Ef admit more flow.
uu vv
0:1
G: uu vv
4
Gf :
Example:
3:5 2
© 2001–4 by Charles E. Leiserson Introduction to Algorithms November 24, 2004 L20.26
Residual networkDefinition. Let f be a flow on G = (V, E). The residual network Gf (V, Ef ) is the graph with strictly positive residual capacities
cf (u, v) = c(u, v) – f (u, v) > 0. Edges in Ef admit more flow.
uu vv
0:1
3:5
G: uu vv
4
2
Gf :
Example:
Lemma. | Ef | ≤ 2| E |. © 2001–4 by Charles E. Leiserson Introduction to Algorithms November 24, 2004 L20.27
Augmenting pathsDefinition. Any path from s to t in Gf is an augmenting path in G with respect to f. The flow value can be increased along an augmentingpath p by c f ( p) = min c f ( v u ).,
,( v u )∈p
© 2001–4 by Charles E. Leiserson Introduction to Algorithms November 24, 2004 L20.28
Augmenting pathsDefinition. Any path from s to t in Gf is an augmenting path in G with respect to f. The flow value can be increased along an augmentingpath p by c f ( p) = min c f ( v u ).,
,( v u )∈p
ss
3:5
G: 2:6 0:2
tt
2:5Ex.:
c5:5 2:3
f (p) = 2 2 4 7 2 3
ssGf : tt 3 2 1 2
© 2001–4 by Charles E. Leiserson Introduction to Algorithms November 24, 2004 L20.29
Max-flow, min-cut theorem
Theorem. The following are equivalent:1. f is a maximum flow. 2. Gf contains no augmenting paths.3. | f | = c(S, T) for some cut (S, T) of G.
© 2001–4 by Charles E. Leiserson Introduction to Algorithms November 24, 2004 L20.30
Max-flow, min-cut theorem
Theorem. The following are equivalent:1. f is a maximum flow. 2. Gf contains no augmenting paths.3. | f | = c(S, T) for some cut (S, T) of G.
Proof (and algorithms). Next time.
© 2001–4 by Charles E. Leiserson Introduction to Algorithms November 24, 2004 L20.31
Introduction to Algorithms6.046J/18.401J
LECTURE 21 Network Flow II • Max-flow, min-cut theorem • Ford-Fulkerson algorithm
and analysis • Edmonds-Karp algorithm
and analysis • Best algorithms to date
Prof. Charles E. Leiserson
Recall from Lecture 22• Flow value: | f | = f (s, V).• Cut: Any partition (S, T) of V such that s ∈ S
and t ∈ T. • Lemma. | f | = f (S, T) for any cut (S, T). • Corollary. | f | ≤ c(S, T) for any cut (S, T).• Residual graph: The graph Gf = (V, Ef ) with
strictly positive residual capacities cf (u, v) = c(u, v) – f (u, v) > 0.
• Augmenting path: Any path from s to t in Gf . • Residual capacity of an augmenting path:
,c f ( p) = min c f ( v u ). ,( v u )∈p
© 2001–4 by Charles E. Leiserson Introduction to Algorithms November 29, 2004 L21.2
Max-flow, min-cut theorem
Theorem. The following are equivalent:1. | f | = c(S, T) for some cut (S, T). 2. f is a maximum flow. 3. f admits no augmenting paths.
© 2001–4 by Charles E. Leiserson Introduction to Algorithms November 29, 2004 L21.3
Max-flow, min-cut theorem
Theorem. The following are equivalent:1. | f | = c(S, T) for some cut (S, T). 2. f is a maximum flow. 3. f admits no augmenting paths. Proof. (1) ⇒ (2): Since | f | ≤ c(S, T) for any cut (S, T) (by the corollary from Lecture 22), the assumption that | f | = c(S, T) implies that f is a maximum flow.
© 2001–4 by Charles E. Leiserson Introduction to Algorithms November 29, 2004 L21.4
Max-flow, min-cut theorem
Theorem. The following are equivalent:1. | f | = c(S, T) for some cut (S, T). 2. f is a maximum flow. 3. f admits no augmenting paths. Proof. (1) ⇒ (2): Since | f | ≤ c(S, T) for any cut (S, T) (by the corollary from Lecture 22), the assumption that | f | = c(S, T) implies that f is a maximum flow. (2) ⇒ (3): If there were an augmenting path, theflow value could be increased, contradicting the maximality of f.
© 2001–4 by Charles E. Leiserson Introduction to Algorithms November 29, 2004 L21.5
Proof (continued)(3) ⇒ (1): Suppose that f admits no augmenting paths. Define S = v ∈ V : there exists a path in Gf from s to v, and let T = V – S. Observe that s ∈ S and t ∈ T, and thus (S, T) is a cut. Consider any vertices u ∈ S and v ∈ T.
ss uu vv S Tpath in Gf
We must have cf (u, v) = 0, since if cf (u, v) > 0, then v ∈ S, not v ∈ T as assumed. Thus, f (u, v) = c(u, v), since cf (u, v) = c(u, v) –
S, T) = cf (u, v). Summing over all u ∈ S and v ∈ T
yields f ( (S, T), and since | f | = f (S, T), the theorem follows. © 2001–4 by Charles E. Leiserson Introduction to Algorithms November 29, 2004 L21.6
Ford-Fulkerson max-flow algorithm
Algorithm:f [u, v] ← 0 for all u, v ∈ V while an augmenting path p in G wrt f exists
do augment f by cf (p)
© 2001–4 by Charles E. Leiserson Introduction to Algorithms November 29, 2004 L21.7
Ford-Fulkerson max-flow algorithm
Algorithm:f [u, v] ← 0 for all u, v ∈ V while an augmenting path p in G wrt f exists
do augment f by cf (p) Can be slow:
ss tt
109 109
109
1
109
G:
© 2001–4 by Charles E. Leiserson Introduction to Algorithms November 29, 2004 L21.8
Ford-Fulkerson max-flow algorithm
Algorithm:f [u, v] ← 0 for all u, v ∈ V while an augmenting path p in G wrt f exists
do augment f by cf (p) Can be slow:
ss tt
0:109 0:109
0:109
0:1
0:109
G:
© 2001–4 by Charles E. Leiserson Introduction to Algorithms November 29, 2004 L21.9
Ford-Fulkerson max-flow algorithm
Algorithm:f [u, v] ← 0 for all u, v ∈ V while an augmenting path p in G wrt f exists
do augment f by cf (p) Can be slow:
ss tt
0:109 0:109
0:109
0:1
0:109
G:
© 2001–4 by Charles E. Leiserson Introduction to Algorithms November 29, 2004 L21.10
Ford-Fulkerson max-flow algorithm
Algorithm:f [u, v] ← 0 for all u, v ∈ V while an augmenting path p in G wrt f exists
do augment f by cf (p) Can be slow:
ss tt
1:109 0:109
1:109
1:1
0:109
G:
© 2001–4 by Charles E. Leiserson Introduction to Algorithms November 29, 2004 L21.11
Ford-Fulkerson max-flow algorithm
Algorithm:f [u, v] ← 0 for all u, v ∈ V while an augmenting path p in G wrt f exists
do augment f by cf (p) Can be slow:
G: ss tt
1:109 0:109
1:109
1:1
0:109
© 2001–4 by Charles E. Leiserson Introduction to Algorithms November 29, 2004 L21.12
Ford-Fulkerson max-flow algorithm
Algorithm:f [u, v] ← 0 for all u, v ∈ V while an augmenting path p in G wrt f exists
do augment f by cf (p) Can be slow:
G: ss tt
1:109 1:109
1:109
0:1
1:109
© 2001–4 by Charles E. Leiserson Introduction to Algorithms November 29, 2004 L21.13
Ford-Fulkerson max-flow algorithm
Algorithm:f [u, v] ← 0 for all u, v ∈ V while an augmenting path p in G wrt f exists
do augment f by cf (p) Can be slow:
ss tt
1:109 1:109
1:109
0:1
1:109
G:
© 2001–4 by Charles E. Leiserson Introduction to Algorithms November 29, 2004 L21.14
Ford-Fulkerson max-flow algorithm
Algorithm:f [u, v] ← 0 for all u, v ∈ V while an augmenting path p in G wrt f exists
do augment f by cf (p) Can be slow:
ss tt
2:109 1:109
2:109
1:1
1:109
G:
© 2001–4 by Charles E. Leiserson Introduction to Algorithms November 29, 2004 L21.15
Ford-Fulkerson max-flow algorithm
Algorithm:f [u, v] ← 0 for all u, v ∈ V while an augmenting path p in G wrt f exists
do augment f by cf (p) Can be slow:
ss tt
2:109 1:109
2:109
1:1
1:109
G:
2 billion iterations on a graph with 4 vertices! © 2001–4 by Charles E. Leiserson Introduction to Algorithms November 29, 2004 L21.16
Edmonds-Karp algorithm
G
Edmonds and Karp noticed that many people’s implementations of Ford-Fulkerson augment along a breadth-first augmenting path: a shortest path in
f from s to t where each edge has weight 1. These implementations would always run relatively fast. Since a breadth-first augmenting path can be found in O(E) time, their analysis, which provided the first polynomial-time bound on maximum flow, focuses on bounding the number of flow augmentations. (In independent work, Dinic also gave polynomial-time bounds.)
© 2001–4 by Charles E. Leiserson Introduction to Algorithms November 29, 2004 L21.17
Monotonicity lemmaLemma. Let δ(v) = δf (s, v) be the breadth-first distance from s to v in Gf . During the Edmonds-Karp algorithm, δ(v) increases monotonically.
© 2001–4 by Charles E. Leiserson Introduction to Algorithms November 29, 2004 L21.18
Monotonicity lemmaLemma. Let δ(v) = δf (s, v) be the breadth-first distance from s to v in Gf . During the Edmonds-Karp algorithm, δ(v) increases monotonically. Proof. Suppose that augmenting a flow f on G produces a new flow f ′. Let δ′(v) = δf ′(s, v). We’ll show δ′(v) ≥ δ(v) by induction on δ′(v). For the base case, δ′(v) = 0 implies v = s, and since δ(s) = 0, we have δ′(v) ≥ δ(v). For the inductive case, consider a breadth-first path s → L → u → v in Gf ′. We must have δ′(v) = δ′(u) + 1, since subpaths of shortest paths are shortest paths. Hence, we have δ′(u) ≥ δ(u) by induction, because δ′(v) > δ′(u). Certainly, (u, v) ∈ Ef ′.
© 2001–4 by Charles E. Leiserson Introduction to Algorithms November 29, 2004 L21.19
Proof of Monotonicity Lemma — Case 1
Consider two cases depending on whether (u, v) ∈ Ef .
Case 1: (u, v) ∈ Ef .
We have δ(v) ≤ δ(u) + 1 (triangle inequality)
≤ δ′(u) + 1 (induction) = δ′(v) (breadth-first path),
and thus monotonicity of δ(v) is established.
© 2001–4 by Charles E. Leiserson Introduction to Algorithms November 29, 2004 L21.20
Proof of Monotonicity Lemma — Case 2
Case: (u, v) ∉ Ef . Since (u, v) ∈ Ef ′ , the augmenting path p that produced f ′ from f must have included (v, u). Moreover, p is a breadth-first path in Gf :
p = s → L → v → u → L → t .Thus, we have
δ(v) = δ(u) – 1 (breadth-first path) ≤ δ′(u) – 1 (induction) = δ′(v) – 2 (breadth-first path) < δ′(v) ,
thereby establishing monotonicity for this case, too. © 2001–4 by Charles E. Leiserson Introduction to Algorithms November 29, 2004 L21.21
Counting flow augmentationsTheorem. The number of flow augmentations in the Edmonds-Karp algorithm (Ford-Fulkerson with breadth-first augmenting paths) is O(VE).
© 2001–4 by Charles E. Leiserson Introduction to Algorithms November 29, 2004 L21.22
Counting flow augmentationsTheorem. The number of flow augmentations in the Edmonds-Karp algorithm (Ford-Fulkerson with breadth-first augmenting paths) is O(VE). Proof. Let p be an augmenting path, and suppose that we have cf (u, v) = cf (p) for edge (u, v) ∈ p. Then, we say that (u, v) is critical, and it disappears from the residual graph after flow augmentation.
© 2001–4 by Charles E. Leiserson Introduction to Algorithms November 29, 2004 L21.23
Counting flow augmentationsTheorem. The number of flow augmentations in the Edmonds-Karp algorithm (Ford-Fulkerson with breadth-first augmenting paths) is O(VE). Proof. Let p be an augmenting path, and suppose that we have cf (u, v) = cf (p) for edge (u, v) ∈ p. Then, we say that (u, v) is critical, and it disappears from the residual graph after flow augmentation. Example: cf (p) = 2
2 4 7 2 3
ssGf : tt 3 2 1 2
© 2001–4 by Charles E. Leiserson Introduction to Algorithms November 29, 2004 L21.24
Counting flow augmentations(continued)
The first time an edge (u, v) is critical, we have δ(v) = δ(u) + 1, since p is a breadth-first path. We must wait until (v, u) is on an augmenting path before (u, v) can be critical again. Let δ′ be the distance function when (v, u) is on an augmenting path. Then, we have
δ′(u) = δ′(v) + 1 (breadth-first path) ≥ δ(v) + 1 (monotonicity) = δ(u) + 2 (breadth-first path).
© 2001–4 by Charles E. Leiserson Introduction to Algorithms November 29, 2004 L21.25
Counting flow augmentations(continued)
The first time an edge (u, v) is critical, we have δ(v) = δ(u) + 1, since p is a breadth-first path. We must wait until (v, u) is on an augmenting path before (u, v) can be critical again. Let δ′ be the distance function when (v, u) is on an augmenting path. Then, we have
δ′(u) = δ′(v) + 1 (breadth-first path) ≥ δ(v) + 1 (monotonicity) = δ(u) + 2 (breadth-first path).
Example: uu
vv ss tt
© 2001–4 by Charles E. Leiserson Introduction to Algorithms November 29, 2004 L21.26
Counting flow augmentations(continued)
The first time an edge (u, v) is critical, we have δ(v) = δ(u) + 1, since p is a breadth-first path. We must wait until (v, u) is on an augmenting path before (u, v) can be critical again. Let δ′ be the distance function when (v, u) is on an augmenting path. Then, we have
δ′(u) = δ′(v) + 1 (breadth-first path) ≥ δ(v) + 1 (monotonicity) = δ(u) + 2 (breadth-first path).
ss uu
vv tt
δ(u) = 5
δ(v) = 6
Example:
© 2001–4 by Charles E. Leiserson Introduction to Algorithms November 29, 2004 L21.27
Counting flow augmentations(continued)
The first time an edge (u, v) is critical, we have δ(v) = δ(u) + 1, since p is a breadth-first path. We must wait until (v, u) is on an augmenting path before (u, v) can be critical again. Let δ′ be the distance function when (v, u) is on an augmenting path. Then, we have
δ′(u) = δ′(v) + 1 (breadth-first path) ≥ δ(v) + 1 (monotonicity) = δ(u) + 2 (breadth-first path).
Example: δ(u) = 5
tt uu
vv ss
δ(v) = 6 © 2001–4 by Charles E. Leiserson Introduction to Algorithms November 29, 2004 L21.28
Counting flow augmentations(continued)
The first time an edge (u, v) is critical, we have δ(v) = δ(u) + 1, since p is a breadth-first path. We must wait until (v, u) is on an augmenting path before (u, v) can be critical again. Let δ′ be the distance function when (v, u) is on an augmenting path. Then, we have
δ′(u) = δ′(v) + 1 (breadth-first path) ≥ δ(v) + 1 (monotonicity) = δ(u) + 2 (breadth-first path).
ss uu
vv tt
δ(u) ≥ 7
δ(v) ≥ 6
Example:
© 2001–4 by Charles E. Leiserson Introduction to Algorithms November 29, 2004 L21.29
Counting flow augmentations(continued)
The first time an edge (u, v) is critical, we have δ(v) = δ(u) + 1, since p is a breadth-first path. We must wait until (v, u) is on an augmenting path before (u, v) can be critical again. Let δ′ be the distance function when (v, u) is on an augmenting path. Then, we have
δ′(u) = δ′(v) + 1 (breadth-first path) ≥ δ(v) + 1 (monotonicity) = δ(u) + 2 (breadth-first path).
Example: δ(u) ≥ 7
tt uu
vv ss
δ(v) ≥ 6 © 2001–4 by Charles E. Leiserson Introduction to Algorithms November 29, 2004 L21.30
Counting flow augmentations(continued)
The first time an edge (u, v) is critical, we have δ(v) = δ(u) + 1, since p is a breadth-first path. We must wait until (v, u) is on an augmenting path before (u, v) can be critical again. Let δ′ be the distance function when (v, u) is on an augmenting path. Then, we have
δ′(u) = δ′(v) + 1 (breadth-first path) ≥ δ(v) + 1 (monotonicity) = δ(u) + 2 (breadth-first path).
ss uu
vv tt
δ(u) ≥ 7
δ(v) ≥ 8
Example:
© 2001–4 by Charles E. Leiserson Introduction to Algorithms November 29, 2004 L21.31
Running time of Edmonds-Karp
Distances start out nonnegative, never decrease, and are at most |V| – 1 until the vertex becomes unreachable. Thus, (u, v) occurs as a critical edge O(V) times, because δ(v) increases by at least 2 between occurrences. Since the residual graph contains O(E) edges, the number of flow augmentations is O(V E).
© 2001–4 by Charles E. Leiserson Introduction to Algorithms November 29, 2004 L21.32
Running time of Edmonds-Karp
Distances start out nonnegative, never decrease, and are at most |V| – 1 until the vertex becomes unreachable. Thus, (u, v) occurs as a critical edge O(V) times, because δ(v) increases by at least 2 between occurrences. Since the residual graph contains O(E) edges, the number of flow augmentations is O(V E).
Corollary. The Edmonds-Karp maximum-flow algorithm runs in O(V E2) time.
© 2001–4 by Charles E. Leiserson Introduction to Algorithms November 29, 2004 L21.33
Running time of Edmonds-Karp
Distances start out nonnegative, never decrease, and are at most |V| – 1 until the vertex becomes unreachable. Thus, (u, v) occurs as a critical edge O(V) times, because δ(v) increases by at least 2 between occurrences. Since the residual graph contains O(E) edges, the number of flow augmentations is O(V E).
Corollary. The Edmonds-Karp maximum-flow algorithm runs in O(V E2) time. Proof. Breadth-first search runs in O(E) time, and all other bookkeeping is O(V) per augmentation.
© 2001–4 by Charles E. Leiserson Introduction to Algorithms November 29, 2004 L21.34
Best to date
• The asymptotically fastest algorithm to date for maximum flow, due to King, Rao, and Tarjan, runs in O(V E logE/(V lg V)V) time.
• If we allow running times as a function of edge weights, the fastest algorithm for maximum flow, due to Goldberg and Rao, runs in time
O(minV 2/3, E 1/2 ⋅ E lg (V 2/E + 2) ⋅ lg C), where C is the maximum capacity of any edge in the graph.
© 2001–4 by Charles E. Leiserson Introduction to Algorithms November 29, 2004 L21.35
Introduction to Algorithms6.046J/18.401J
Lecture 22Prof. Piotr Indyk
Today
• String matching problems • HKN Evaluations (last 15 minutes) • Graded Quiz 2 (outside)
© Piotr Indyk Introduction to Algorithms December 1, 2004 L22.2
String Matching
• Input: Two strings T[1…n] and P[1…m],containing symbols from alphabet Σ. E.g. :–Σ=a,b,…,z– T[1…18]=“to be or not to be” – P[1..2]=“be”
• Goal: find all “shifts” 0≤ s ≤n-m such that T[s+1…s+m]=P E.g. 3, 16
© Piotr Indyk Introduction to Algorithms December 1, 2004 L22.3
Simple Algorithm
for s ← 0 to n-m
Match ← 1
for j ← 1 to m
if T[s+j] ≠P[j] then Match ← 0 exit loop
if Match=1 then output s
© Piotr Indyk Introduction to Algorithms December 1, 2004 L22.4
Results
• Running time of the simple algorithm: – Worst-case: O(nm) – Average-case (random text): O(n)
• Ts= time spent on checking shift s • E[Ts] ≤ 2 • E [∑sTs] = ∑s E[Ts] = O(n)
© Piotr Indyk Introduction to Algorithms December 1, 2004 L22.5
Worst-case
• Is it possible to achieve O(n) for any input ?
– Knuth-Morris-Pratt’77: deterministic – Karp-Rabin’81: randomized
© Piotr Indyk Introduction to Algorithms December 1, 2004 L22.6
Karp-Rabin Algorithm
• A very elegant use of an idea that we have encounteredbefore, namely…
HASHING ! • Idea:
– Hash all substrings T[1…m], T[2…m+1], …, T[m-n+1…n]
– Hash the pattern P[1…m] – Report the substrings that hash to the same value as P
• Problem: how to hash n-m substrings, each of length m, inO(n) time ?
© Piotr Indyk Introduction to Algorithms December 1, 2004 L22.7
Attempt 0
• In Lecture 7, we have seen
ha(x)=∑i aixi mod q
where a=(a1,…,ar) , x=(x1,…,xr)• To implement it, we would need to compute
ha( T[s…s+m-1] )=∑i ai T[s+i] mod q for s=0…n-m
• How to compute it in O(n) time ? • A big open problem!
© Piotr Indyk Introduction to Algorithms December 1, 2004 L22.8
Attempt 1
• Assume Σ=0,1• Think about each Ts=T[s+1…s+m] as a
number in binary representation, i.e., ts=T[s+1]2m-1+T[s+2]2m-2+…+T[s+m]20
• Find a fast way of computing ts+1 given ts • Output all s such that ts is equal to the
number p represented by P
© Piotr Indyk Introduction to Algorithms December 1, 2004 L22.9
The great formula
• How to transform ts=T[s+1]2m-1+T[s+2]2m-2+…+T[s+m]20
into ts+1=T[s+2]2m-1+T[s+3]2m-2+…+T[s+m+1]20 ?
• Three steps:– Subtract T[s+1]2m-1
– Multiply by 2 (i.e., shift the bits by oneposition)
– Add T[s+m+1]20
• Therefore: ts+1= (ts- T[s+1]2m-1)*2 + T[s+m+1]20
© Piotr Indyk Introduction to Algorithms December 1, 2004 L22.10
Algorithm
ts+1= (ts- T[s+1]2m-1)*2 + T[s+m+1]20
• Can compute ts+1 from ts using 3 arithmetic operations
• Therefore, we can compute all t0,t1,…,tn-musing O(n) arithmetic operations
• We can compute a number corresponding toP using O(m) arithmetic operations
• Are we done ?
© Piotr Indyk Introduction to Algorithms December 1, 2004 L22.11
Problem
• To get O(n) time, we would need to perform each arithmetic operation in O(1) time
• However, the arguments are m-bit long !• If m large, it is unreasonable to assume that
operations on such big numbers can be done in O(1) time
• We need to reduce the number range to something more managable
© Piotr Indyk Introduction to Algorithms December 1, 2004 L22.12
Attempt 2: Hashing
• We will instead compute t’s=T[s+1]2m-1+T[s+2]2m-2+…+T[s+m]20 mod q where q is an “appropriate” prime number
• One can still compute t’s+1 from t’s : t’s+1= (t’s- T[s+1]2m-1)*2+T[s+m+1]20 mod q
• If q is not large, i.e., has O(log n) bits, we can compute all t’s (and p’) in O(n) time
© Piotr Indyk Introduction to Algorithms December 1, 2004 L22.13
Problem
• Unfortunately, we can have false positives, i.e., Ts≠P but ts mod q = p mod q
• Need to use a random q
• We will show that the probability of a false positive is small → randomized algorithm
© Piotr Indyk Introduction to Algorithms December 1, 2004 L22.14
False positives
• Consider any ts≠p. We know that both numbers are in the range 0…2m-1
• How many primes q are there such that ts mod q = p mod q ≡ (ts-p) =0 mod q ?
• Such prime has to divide x=(ts-p) ≤ 2m
e1p2e2…pk
ek• Represent x=p1 , pi prime, ei≥1What is the largest possible value of k ?
– Since 2 ≤ pi , we have x ≥ 2k
– But x ≤ 2m
– k ≤ m• There are ≤ m primes dividing x
© Piotr Indyk Introduction to Algorithms December 1, 2004 L22.15
Algorithm
• Algorithm: – Let ∏ be a set of 2nm primes, each having O(log n) bits – Choose q uniformly at random from ∏ – Compute t0 mod q, t1 mod q, …., and p mod q – Report s such that ts mod q = p mod q
• Analysis: – For each s, the probability that Ts≠P but
ts mod q =p mod qis at most m/2nm = 1/2n
– The probability of any false positive is at most (n-m)/2n ≤ 1/2
© Piotr Indyk Introduction to Algorithms December 1, 2004 L22.16
“Details”
• How do we know that such ∏ exists ? (That is, a set of 2nm primes, each having O(log n) bits)
• How do we choose a random prime from ∏ in O(n) time ?
© Piotr Indyk Introduction to Algorithms December 1, 2004 L22.17
Prime density
• Primes are “dense”. I.e., if PRIMES(N) is the set of primes smaller than N, thenasymptotically
|PRIMES(N)|/N ~ 1/ln N• If N large enough, then
|PRIMES(N)| ≥ N/(2ln N)
• Proof: Trust me.
© Piotr Indyk Introduction to Algorithms December 1, 2004 L22.18
Prime density continued
• Set N=C mn ln(mn) • There exists C=O(1) such that
N/(2ln N) ≥ 2mn (Note: for such N we have PRIMES(N) ≥ 2mn )
• Proof:C mn ln(mn) / [2 ln(C mn ln(mn)) ]
≥ C mn ln(mn) / [2 ln(C (mn)2) ] = C mn ln(mn) / 4[ ln(C) + ln(mn)] • All elements of PRIMES(N) are log N = O(log n)
bits long
© Piotr Indyk Introduction to Algorithms December 1, 2004 L22.19
Prime selection
• Still need to find a random element of PRIMES(N)
• Solution: – Choose a random element from 1 … N – Check if it is prime – If not, repeat
© Piotr Indyk Introduction to Algorithms December 1, 2004 L22.20
Prime selection analysis
• A random element q from 1…N is prime with probability ~1/ln N
• We can check if q is prime in time polynomial in log N : – Randomized: Rabin, Solovay-Strassen in 1976
– Deterministic: Agrawal et al in 2002 • Therefore, we can generate random prime q in
o(n) time
© Piotr Indyk Introduction to Algorithms December 1, 2004 L22.21
Final Algorithm
• Set N=C mn ln(mn)• Repeat
– Choose q uniformly at random from1…N
• Until q is prime• Compute t0 mod q, t1 mod q, …., and p mod
q • Report s such that ts mod q = p mod q
© Piotr Indyk Introduction to Algorithms December 1, 2004 L22.22
Introduction to Algorithms6.046J/18.401J
Lecture 24Prof. Piotr Indyk
Dealing with Hard Problems
• What to do if: – Divide and conquer – Dynamic programming
– Greedy
– Linear Programming/Network Flows
– …
does not give a polynomial time algorithm?
© Piotr Indyk Introduction to Algorithms December 8, 2004 L24.2
Dealing with Hard Problems
• Solution I: Ignore the problem
– Can’t do it ! There are thousands of problems for which we do not know polynomial time algorithms
– For example: • Traveling Salesman Problem (TSP) • Set Cover
© Piotr Indyk Introduction to Algorithms December 8, 2004 L24.3
Traveling Salesman Problem
• Traveling SalesmanProblem (TSP)– Input: undirected graph
with lengths on edges– Output: shortest cycle
that visits each vertex exactly once
• Best known algorithm:O(n 2n) time.
© Piotr Indyk Introduction to Algorithms December 8, 2004 L24.4
Set Covering
Bank robbery problem:• Set Cover: • X=plan, shoot, safe,
∪– Input: subsets S1…Sn of X, drive, scary
i Si = X, |X|=m • Sets:– Output: C ⊆1…n , such – SJoe =plan, safe
that ∪i∈C Si = X, and |C| – SJim=shoot, scary, minimal drive
• Best known algorithm: – …. O(2n m) time(?)
© Piotr Indyk Introduction to Algorithms December 8, 2004 L24.5
Dealing with Hard Problems
• Exponential time algorithms for small inputs. E.g., (100/99)n time is not bad for n < 1000.
• Polynomial time algorithms for some (e.g., average-case) inputs
• Polynomial time algorithms for all inputs, but which return approximate solutions
© Piotr Indyk Introduction to Algorithms December 8, 2004 L24.6
Approximation Algorithms
• An algorithm A is ρ-approximate, if, on any inputof size n: – The cost CA of the solution produced by the
algorithm, and – The cost COPT of the optimal solution are such that CA ≤ ρ COPT
• We will see: – 2-approximation algorithm for TSP in the plane – ln(m)-approximation algorithm for Set Cover
© Piotr Indyk Introduction to Algorithms December 8, 2004 L24.7
Comments on Approximation
• “ CA ≤ ρ COPT “ makes sense only forminimization problems
• For maximization problems, replace by ““CA ≥ 1/ρ COPT
• Additive approximation “CA ≤ ρ + COPT “ also makes sense, although difficult to achieve
© Piotr Indyk Introduction to Algorithms December 8, 2004 L24.8
2-approximation for TSP
• Compute MST T – An edge between any pair of points
– Weight = distance between endpoints
• Compute a tree-walk W of T – Each edge visited twice
• Convert W into a cycle C using shortcuts
© Piotr Indyk Introduction to Algorithms December 8, 2004 L24.9
2-approximation: Proof
• Let COPT be the optimal cycle
• Cost(T) ≤ Cost(COPT ) – Removing an edge from C gives a spanning
tree, T is a spanning tree of minimum cost
• Cost(W) = 2 Cost(T) – Each edge visited twice
• Cost(C) ≤ Cost(W) – Triangle inequality
⇒ Cost(C) ≤ 2 Cost(COPT )
© Piotr Indyk Introduction to Algorithms December 8, 2004 L24.10
Approximation for Set Cover
Greedy algorithm: • Initialize C=∅
• Repeat until all elements are covered:– Choose Si which contains largest number
of yet-not-covered elements – Add i to C – Mark all elements in Si as covered
© Piotr Indyk Introduction to Algorithms December 8, 2004 L24.11
Greedy Algorithm: Example
• X=1,2,3,4,5,6
• Sets: – S1=1,2 – S2=3,4 – S3=5,6 – S4=1,3,5
• Algorithm picks C=4,1,2,3 • Not optimal!
© Piotr Indyk Introduction to Algorithms December 8, 2004 L24.12
ln(m)-approximation
• Notation:– COPT = optimal cover– k=|COPT |
• Fact: At any iteration of the algorithm, there exists Sjwhich contains at ≥ 1/k fraction of yet-not-coveredelements
• Proof: by contradiction. – If all sets cover <1/k fraction of yet-not-covered
elements, there is no way to cover them using k sets – But COPT does that !
• Therefore, at each iteration greedy covers ≥1/k fraction of yet-not-covered elements
© Piotr Indyk Introduction to Algorithms December 8, 2004 L24.13
ln(m)-approximation
• Let ui be the number of yet-not-covered elementsat the end of step i=0,1,2,…
• We haveui+1 ≤ ui (1-1/k)
u0=m • Therefore, after t=k ln m steps, we have
ut ≤ u0 (1-1/k)t ≤ m (1-1/k)k ln m < m 1/eln m =1 • I.e., all elements are covered by the k ln m sets
chosen by greedy algorithm • Opt size is k ⇒ greedy is ln(m)-approximate
© Piotr Indyk Introduction to Algorithms December 8, 2004 L24.14
Approximation Algorithms
• Very rich area– Algorithms use greedy, linear
programming, dynamic programming • E.g., 1.01-approximate TSP in the plane
– Sometimes can show that approximating a problem is as hard as finding exactsolution ! • E.g., 0.99 ln(m)-approximate Set Cover
© Piotr Indyk Introduction to Algorithms December 8, 2004 L24.15