BİM 202 ALGORITHMS
description
Transcript of BİM 202 ALGORITHMS
ÖĞR. GÖR. ZEYNEP TURGUT
BİM 202 ALGORITHMS
References - Books
Introduction to AlgorithmsCormen, Leiserson and Rivest and Stein
Data Structures & Algorithm Analysis in C++Mark Allen Weiss
Recommended Readings:Algoritmalar Prof. Dr. Vasif Vagifoglu NabiyevVeri Yapıları ve Algoritmalar
Dr. Rifat Çölkesen
Grading Guidelines
Two Exams: 20% midterm, 60% final.
%20 projects & homeworks & quizzes & attitude to the lesson.
Minimum 70% attendance to lesson required to clear the course.
Topics
Introduction to AlgorithmsAlgorithm AnalysisAlgorithm TypesSorting AlgorithmsSearching AlgorithmsGraph AlgorithmsTree AlgorithmsNP-Complete Problems
WARNING: This lecture contains mathematical content that may be shocking to some students.
Good, better, best. Never let it rest. ‘Till your good is better and your better is best.Saint Jerome
The Computer as a Tool
Much like the microscope does not define biology or the test tube does not define chemistry, the computer doesn't define Computer Science.
The computer is a tool by which Computer Scientists accomplish their goals – to solve problems.
What is Computer Science?
NOT about coding or hardware or software!
Computer Science is about PROBLEM SOLVING
Computer Science is about DEVELOPING ALGORITHMS to solve complex problems
Problem Solving: Main Steps
1. Problem definition2. Algorithm design / Algorithm specification3. Algorithm analysis4. Implementation5. Testing6. [Maintenance]
1. Problem Definition
What is the task to be accomplished? Calculate the average of the grades for a given
student Understand the talks given out by politicians and
translate them in Chinese
What are the time / space / speed / performance requirements ?
2. Algorithm Design / Specifications
Algorithm: Finite set of instructions that, if followed, accomplishes a particular task.
Describe: in natural language / pseudo-code / diagrams / etc.
Criteria to follow: Input: Zero or more quantities (externally produced) Output: One or more quantities Definiteness: Clarity, precision of each instruction Finiteness: The algorithm has to stop after a finite
(may be very large) number of steps Effectiveness: Each instruction has to be basic enough
and feasible Understand speech Translate to Chinese
Algorithm Questions
Computer Scientists ask themselves four critical questions when they evaluate algorithms …
Does the algorithm solve the stated problem?
Is the algorithm well-defined?
Does the algorithm produce an output?
Does the algorithm end in a reasonable length of time?
Developing an Algorithm
1. Identify the Inputs
2. Identify the Processes
3. Identify the Outputs
4. Develop a HIPO Chart
5. Identify modules
2.1. Identify the Inputs
What data do I need?
How will I get the data?
In what format will the data be?
2.2. Identify the Processes
How can I manipulate data to produce meaningful results?
Data vs. Information
2.3. Identify the Outputs
What outputs do I need to return to the user?
What format should the outputs take?
2.4. Develop a HIPO Chart
Hierarchy of Inputs Processes & Outputs
2.5. Identify Relevant Modules
How can I break larger problems into smaller, more manageable pieces?
What inputs do the modules need?
What processes need to happen in the modules?
What outputs are produced by the modules?
Summary
The goal of Computer Science is to develop sound algorithms for solving complex problems.
An algorithm is a well-developed, detailed approach for solving a problem.
Summary
To develop an algorithm:1. Identify the inputs2. Identify the processes3. Identify the outputs4. Develop a HIPO chart5. Identify relevant modules
Space complexity How much space is required
Time complexity How much time does it take to run the algorithm
Often, we deal with estimates!
3. Algorithm Analysis
4,5,6: Implementation, Testing, Maintainance
Implementation Decide on the programming language to use
C, C++, Lisp, Java, Perl, Prolog, assembly, etc. , etc. Write clean, well documented code
Test, test, test
Integrate feedback from users, fix bugs, ensure compatibility across different versions Maintenance
Space Complexity
Space complexity = The amount of memory required by an algorithm to run to completion [Core dumps = the most often encountered cause is
“memory leaks” – the amount of memory required larger than the memory available on a given system]
Some algorithms may be more efficient if data completely loaded into memory Need to look also at system limitations E.g. Classify 2GB of text in various categories [politics,
tourism, sport, natural disasters, etc.] – can I afford to load the entire collection?
Space Complexity (cont’d)
1. Fixed part: The size required to store certain data/variables, that is independent of the size of the problem:- e.g. name of the data collection- same size for classifying 2GB or 1MB of texts
2. Variable part: Space needed by variables, whose size is dependent on the size of the problem:- e.g. actual text - load 2GB of text VS. load 1MB of text
Space Complexity (cont’d)
S(P) = c + S(instance characteristics) c = constant
Example:void float sum (float* a, int n) {
float s = 0; for(int i = 0; i<n; i++) { s+ = a[i]; } return s;}Space? one word for n, one for a [passed by reference!], one
for i constant space!
Time Complexity
Often more important than space complexity space available (for computer programs!) tends to be larger
and larger time is still a problem for all of us
3-4GHz processors on the market still … researchers estimate that the computation of various
transformations for 1 single DNA chain for one single protein on 1 TerraHZ computer would take about 1 year to run to completion
Algorithms running time is an important issue
Running Time
Problem: prefix averages Given an array X Compute the array A such that A[i] is the average of
elements X[0] … X[i], for i=0..n-1Sol 1
At each step i, compute the element X[i] by traversing the array A and determining the sum of its elements, respectively the average
Sol 2 At each step i update a sum of the elements in the
array A Compute the element X[i] as sum/I
Big question: Which solution to choose?
Running time
Input
1 ms
2 ms
3 ms
4 ms
5 ms
A B C D E F G
worst-case
best-case}average-case?
Suppose the program includes an if-then statement that may execute or not: variable running timeTypically algorithms are measured by their worst case
Experimental Approach
Write a program that implements the algorithm
Run the program with data sets of varying size.
Determine the actual running time using a system call to measure time (e.g. system (date) );
Problems?
Experimental Approach
It is necessary to implement and test the algorithm in order to determine its running time.
Experiments can be done only on a limited set of inputs, and may not be indicative of the running time for other inputs.
The same hardware and software should be used in order to compare two algorithms. – condition very hard to achieve!
Use a Theoretical Approach
Based on high-level description of the algorithms, rather than language dependent implementations
Makes possible an evaluation of the algorithms that is independent of the hardware and software environments
Generality
Algorithm Description
How to describe algorithms independent of a programming language
Pseudo-Code = a description of an algorithm that is more structured than usual prose but less formal than a programming language
(Or diagrams)Example: find the maximum element of an array.
Algorithm arrayMax(A, n):Input: An array A storing n integers.Output: The maximum element in A.currentMax A[0]for i 1 to n -1 doif currentMax < A[i] then currentMax A[i]return currentMax
Pseudo Code
Expressions: use standard mathematical symbols use for assignment ( ? in C/C++) use = for the equality relationship (? in C/C++)
Method Declarations: -Algorithm name(param1, param2) Programming Constructs:
decision structures: if ... then ... [else ..] while-loops while ... do repeat-loops: repeat ... until ... for-loop: for ... do array indexing: A[i]
Methods calls: object method(args) returns: return value
Use comments Instructions have to be basic enough and feasible!
Low Level Algorithm Analysis
Based on primitive operations (low-level computations independent from the programming language)
E.g.:Make an addition = 1 operationCalling a method or returning from a method = 1 operation Index in an array = 1 operationComparison = 1 operation etc.
Method: Inspect the pseudo-code and count the number of primitive operations executed by the algorithm
Example
Algorithm arrayMax(A, n):Input: An array A storing n integers.Output: The maximum element in A.
currentMax A[0]for i 1 to n -1 doif currentMax < A[i] then
currentMax A[i]return currentMax
How many operations ?
Types of Algorithms
37
Algorithm classification
Algorithms that use a similar problem-solving approach can be grouped together
We’ll talk about a classification scheme for algorithms
This classification scheme is neither exhaustive nor disjoint
The purpose is not to be able to classify an algorithm as one type or another, but to highlight the various ways in which a problem can be attacked
38
A short list of categories
Algorithm types we will consider include: Simple recursive algorithms Backtracking algorithms Divide and conquer algorithms Dynamic programming algorithms Greedy algorithms Branch and bound algorithms Brute force algorithms Randomized algorithms
39
Simple recursive algorithms I
A simple recursive algorithm: Solves the base cases directly Recurs with a simpler subproblem Does some extra work to convert the solution to the
simpler subproblem into a solution to the given problem
I call these “simple” because several of the other algorithm types are inherently recursive
40
Example recursive algorithms
To count the number of elements in a list: If the list is empty, return zero; otherwise, Step past the first element, and count the remaining
elements in the list Add one to the result
To test if a value occurs in a list: If the list is empty, return false; otherwise, If the first thing in the list is the given value, return
true; otherwise Step past the first element, and test whether the value
occurs in the remainder of the list
41
L8
Recursive Algorithms
long factorial(int n){if (n<=0) return 1;return n*factorial(n-1);
}
Compute 5!
42
L8
Recursive Algorithms
long factorial(int n){if (n<=0) return 1;return n*factorial(n-1);
}
f(5)=5·f(4)
43
L8
Recursive Algorithms
long factorial(int n){if (n<=0) return 1;return n*factorial(n-1);
}
f(4)=4·f(3)
f(5)=5·f(4)
44
L8
Recursive Algorithms
long factorial(int n){if (n<=0) return 1;return n*factorial(n-1);
}
f(3)=3·f(2)
f(4)=4·f(3)
f(5)=5·f(4)
45
L8
Recursive Algorithms
long factorial(int n){if (n<=0) return 1;return n*factorial(n-1);
}
f(2)=2·f(1)
f(3)=3·f(2)
f(4)=4·f(3)
f(5)=5·f(4)
46
L8
Recursive Algorithms
long factorial(int n){if (n<=0) return 1;return n*factorial(n-1);
}
f(1)=1·f(0)
f(2)=2·f(1)
f(3)=3·f(2)
f(4)=4·f(3)
f(5)=5·f(4)
47
L8
Recursive Algorithms
long factorial(int n){if (n<=0) return 1;return n*factorial(n-1);
}
f(0)=1
f(1)=1·f(0)
f(2)=2·f(1)
f(3)=3·f(2)
f(4)=4·f(3)
f(5)=5·f(4)
48
L8
Recursive Algorithms
long factorial(int n){if (n<=0) return 1;return n*factorial(n-1);
}
1·1=1
f(2)=2·f(1)
f(3)=3·f(2)
f(4)=4·f(3)
f(5)=5·f(4)
49
L8
Recursive Algorithms
long factorial(int n){if (n<=0) return 1;return n*factorial(n-1);
}
2·1=2
f(3)=3·f(2)
f(4)=4·f(3)
f(5)=5·f(4)
50
L8
Recursive Algorithms
long factorial(int n){if (n<=0) return 1;return n*factorial(n-1);
}
3·2=6
f(4)=4·f(3)
f(5)=5·f(4)
51
L8
Recursive Algorithms
long factorial(int n){if (n<=0) return 1;return n*factorial(n-1);
}
4·6=24
f(5)=5·f(4)
52
L8
Recursive Algorithms
long factorial(int n){if (n<=0) return 1;return n*factorial(n-1);
}
5·24=
120
53
L8
Recursive Algorithms
long factorial(int n){if (n<=0) return 1;return n*factorial(n-1);
}
Return 5! = 120
54
Backtracking algorithms
Backtracking algorithms are based on a depth-first recursive search
A backtracking algorithm: Tests to see if a solution has been found, and if so,
returns it; otherwise For each choice that can be made at this point,
Make that choice Recur If the recursion returns a solution, return it
If no choices remain, return failure
55
Example backtracking algorithm
To color a map with no more than four colors: color(Country n):
If all countries have been colored (n > number of countries) return success; otherwise,
For each color c of four colors, If country n is not adjacent to a country that
has been colored c• Color country n with color c• recursively color country n+1• If successful, return success
If loop exits, return failure
56
Backtracking
Suppose you have to make a series of decisions, among various choices, where You don’t have enough information to know what
to choose Each decision leads to a new set of choices Some sequence of choices (possibly more than
one) may be a solution to your problemBacktracking is a methodical way of
trying out various sequences of decisions, until you find one that “works”
57
Solving a maze
Given a maze, find a path from start to finishAt each intersection, you have to decide
between three or fewer choices: Go straight Go left Go right
You don’t have enough information to choose correctly
Each choice leads to another set of choicesOne or more sequences of choices may (or may not)
lead to a solutionMany types of maze problem can be solved with
backtracking
58
Coloring a map
You wish to color a map withnot more than four colors red, yellow, green, blue
Adjacent countries must be indifferent colors
You don’t have enough information to choose colorsEach choice leads to another set of choicesOne or more sequences of choices may (or may not)
lead to a solutionMany coloring problems can be solved with
backtracking
59
Solving a puzzle
In this puzzle, all holes but oneare filled with white pegs
You can jump over one pegwith another
Jumped pegs are removedThe object is to remove all
but the last pegYou don’t have enough information to jump
correctlyEach choice leads to another set of choicesOne or more sequences of choices may (or may
not) lead to a solutionMany kinds of puzzle can be solved with
backtracking
60
Divide and Conquer
A divide and conquer algorithm consists of two parts: Divide the problem into smaller subproblems of the
same type, and solve these subproblems recursively Combine the solutions to the subproblems into a
solution to the original problem Traditionally, an algorithm is only called
“divide and conquer” if it contains at least two recursive calls
61
Examples
Quicksort: Partition the array into two parts (smaller numbers
in one part, larger numbers in the other part) Quicksort each of the parts No additional work is required to combine the two
sorted parts Mergesort:
Cut the array in half, and mergesort each half Combine the two sorted arrays into a single sorted
array by merging them
62
Binary tree lookup
Here’s how to look up something in a sorted binary tree: Compare the key to the value in the root
If the two values are equal, report success If the key is less, search the left subtree If the key is greater, search the right subtree
This is not a divide and conquer algorithm because, although there are two recursive calls, only one is used at each level of the recursion
63
Fibonacci numbers
To find the nth Fibonacci number: If n is zero or one, return one; otherwise, Compute fibonacci(n-1) and fibonacci(n-2) Return the sum of these two numbers
This is an expensive algorithm It requires O(fibonacci(n)) time This is equivalent to exponential time, that is, O(2n)
64
Dynamic programming algorithms
A dynamic programming algorithm remembers past results and uses them to find new results
Dynamic programming is generally used for optimization problems Multiple solutions exist, need to find the “best” one Requires “optimal substructure” and “overlapping
subproblems” Optimal substructure: Optimal solution contains optimal
solutions to subproblems Overlapping subproblems: Solutions to subproblems can
be stored and reused in a bottom-up fashionThis differs from Divide and Conquer, where
subproblems generally need not overlap
65
Fibonacci numbers again
To find the nth Fibonacci number: If n is zero or one, return one; otherwise, Compute, or look up in a table, fibonacci(n-1) and
fibonacci(n-2) Find the sum of these two numbers Store the result in a table and return it
Since finding the nth Fibonacci number involves finding all smaller Fibonacci numbers, the second recursive call has little work to do
The table may be preserved and used again later
66
Greedy algorithms
An optimization problem is one in which you want to find, not just a solution, but the best solution
A “greedy algorithm” sometimes works well for optimization problems
A greedy algorithm works in phases: At each phase: You take the best you can get right now, without
regard for future consequences You hope that by choosing a local optimum at each
step, you will end up at a global optimum
67
Example: Counting moneySuppose you want to count out a certain
amount of money, using the fewest possible bills and coins
A greedy algorithm would do this would be:At each step, take the largest possible bill or coin that does not overshoot Example: To make $6.39, you can choose:
a $5 bill a $1 bill, to make $6 a 25¢ coin, to make $6.25 A 10¢ coin, to make $6.35 four 1¢ coins, to make $6.39
For US money, the greedy algorithm always gives the optimum solution
68
A failure of the greedy algorithmIn some (fictional) monetary system,
“krons” come in 1 kron, 7 kron, and 10 kron coins
Using a greedy algorithm to count out 15 krons, you would get A 10 kron piece Five 1 kron pieces, for a total of 15 krons This requires six coins
A better solution would be to use two 7 kron pieces and one 1 kron piece This only requires three coins
The greedy algorithm results in a solution, but not in an optimal solution
69
Branch and bound algorithms
Branch and bound algorithms are generally used for optimization problems As the algorithm progresses, a tree of subproblems is
formed The original problem is considered the “root problem” A method is used to construct an upper and lower
bound for a given problem At each node, apply the bounding methods
If the bounds match, it is deemed a feasible solution to that particular subproblem
If bounds do not match, partition the problem represented by that node, and make the two subproblems into children nodes
Continue, using the best known feasible solution to trim sections of the tree, until all nodes have been solved or trimmed
70
Example branch and bound algorithm
Traveling salesman problem: A salesman has to visit each of n cities (at least) once each, and wants to minimize total distance traveled
Consider the root problem to be the problem of finding the shortest route through a set of cities visiting each city once
Split the node into two child problems: Shortest route visiting city A first Shortest route not visiting city A first
Continue subdividing similarly as the tree grows
71
Brute force algorithm
A brute force algorithm simply tries all possibilities until a satisfactory solution is found Such an algorithm can be:
Optimizing: Find the best solution. This may require finding all solutions, or if a value for the best solution is known, it may stop when any best solution is found Example: Finding the best path for a traveling
salesman Satisficing: Stop as soon as a solution is found that is
good enough Example: Finding a traveling salesman path that is
within 10% of optimal
72
Improving brute force algorithms
Often, brute force algorithms require exponential time
Various heuristics and optimizations can be used Heuristic: A “rule of thumb” that helps you decide
which possibilities to look at first Optimization: In this case, a way to eliminate certain
possibilities without fully exploring them
73
Randomized algorithms
A randomized algorithm uses a random number at least once during the computation to make a decision
Example: In Quicksort, using a random number to choose a pivot
Example: Trying to factor a large number by choosing random numbers as possible divisors
Simple Sorting Algorithms
75
Bubble sort
Compare each element (except the last one) with its neighbor to the right If they are out of order, swap them This puts the largest element at the very end The last element is now in the correct and final place
Compare each element (except the last two) with its neighbor to the right If they are out of order, swap them This puts the second largest element next to last The last two elements are now in their correct and final
placesCompare each element (except the last three) with
its neighbor to the right Continue as above until you have no unsorted elements on
the left
76
Example of bubble sort
7 2 8 5 4
2 7 8 5 4
2 7 8 5 4
2 7 5 8 4
2 7 5 4 8
2 7 5 4 8
2 5 7 4 8
2 5 4 7 8
2 7 5 4 8
2 5 4 7 8
2 4 5 7 8
2 5 4 7 8
2 4 5 7 8
2 4 5 7 8
(done)
77
Code for bubble sort
public static void bubbleSort(int[] a) { int outer, inner; for (outer = a.length - 1; outer > 0; outer--) { // counting down for (inner = 0; inner < outer; inner++) { // bubbling up if (a[inner] > a[inner + 1]) { // if out of order... int temp = a[inner]; // ...then swap a[inner] = a[inner + 1]; a[inner + 1] = temp; } } }}
78Analysis of bubble sort
for (outer = a.length - 1; outer > 0; outer--) { for (inner = 0; inner < outer; inner++) { if (a[inner] > a[inner + 1]) { // code for swap omitted} } }
Let n = a.length = size of the array The outer loop is executed n-1 times (call it n, that’s
close enough) Each time the outer loop is executed, the inner loop is
executed Inner loop executes n-1 times at first, linearly dropping
to just once On average, inner loop executes about n/2 times for
each execution of the outer loop In the inner loop, the comparison is always done
(constant time), the swap might be done (also constant time)
Result is n * n/2 * k, that is, O(n2/2 + k) = O(n2)
79
Loop invariants
You run a loop in order to change thingsOddly enough, what is usually most important in
understanding a loop is finding an invariant: that is, a condition that doesn’t change
In bubble sort, we put the largest elements at the end, and once we put them there, we don’t move them again The variable outer starts at the last index in the array and
decreases to 0 Our invariant is: Every element to the right of outer is in the
correct place That is, for all j > outer, if i < j, then a[i] <= a[j] When this is combined with the loop exit test, outer == 0, we
know that all elements of the array are in the correct place
80
Selection sort
Given an array of length n, Search elements 0 through n-1 and select the smallest
Swap it with the element in location 0 Search elements 1 through n-1 and select the smallest
Swap it with the element in location 1 Search elements 2 through n-1 and select the smallest
Swap it with the element in location 2 Search elements 3 through n-1 and select the smallest
Swap it with the element in location 3 Continue in this fashion until there’s nothing left to
search
81
Example and analysis of selection sort
The selection sort might swap an array element with itself--this is harmless, and not worth checking for
Analysis: The outer loop executes n-1 times The inner loop executes about n/2
times on average (from n to 2 times) Work done in the inner loop is
constant (swap two array elements) Time required is roughly (n-1)*(n/2) You should recognize this as O(n2)
7 2 8 5 4
2 7 8 5 4
2 4 8 5 7
2 4 5 8 7
2 4 5 7 8
82
Code for selection sort
public static void selectionSort(int[] a) { int outer, inner, min; for (outer = 0; outer < a.length - 1; outer++) { min = outer; for (inner = outer + 1; inner < a.length; inner++) { if (a[inner] < a[min]) { min = inner; } // Invariant: for all i, if outer <= i <= inner, then a[min] <= a[i] }
// a[min] is least among a[outer]..a[a.length - 1] int temp = a[outer]; a[outer] = a[min]; a[min] = temp; // Invariant: for all i <= outer, if i < j then a[i] <= a[j] }}
83
Invariants for selection sort
For the inner loop: This loop searches through the array, incrementing inner
from its initial value of outer+1 up to a.length-1 As the loop proceeds, min is set to the index of the
smallest number found so far Our invariant is:
for all i such that outer <= i <= inner, a[min] <= a[i]For the outer (enclosing) loop:
The loop counts up from outer = 0 Each time through the loop, the minimum remaining
value is put in a[outer] Our invariant is:
for all i <= outer, if i < j then a[i] <= a[j]
84
Insertion sort
The outer loop of insertion sort is: for (outer = 1; outer < a.length; outer++) {...}
The invariant is that all the elements to the left of outer are sorted with respect to one another For all i < outer, j < outer, if i < j then a[i] <= a[j] This does not mean they are all in their final correct place;
the remaining array elements may need to be inserted When we increase outer, a[outer-1] becomes to its left; we
must keep the invariant true by inserting a[outer-1] into its proper place
This means: Finding the element’s proper place Making room for the inserted element (by shifting over other
elements) Inserting the element
85
One step of insertion sort
3 4 7 12 14 14 20 21 33 38 10 55 9 23 28 16
sorted next to be inserted
3 4 7 55 9 23 28 16
10temp
3833212014141210
sorted
less than 10
86
Analysis of insertion sort
We run once through the outer loop, inserting each of n elements; this is a factor of n
On average, there are n/2 elements already sorted The inner loop looks at (and moves) half of these This gives a second factor of n/4
Hence, the time required for an insertion sort of an array of n elements is proportional to n2/4
Discarding constants, we find that insertion sort is O(n2)
87
Summary
Bubble sort, selection sort, and insertion sort are all O(n2)
As we will see later, we can do much better than this with somewhat more complicated sorting algorithms
Within O(n2), Bubble sort is very slow, and should probably never be used
for anything Selection sort is intermediate in speed Insertion sort is usually the fastest of the three--in fact, for
small arrays (say, 10 or 15 elements), insertion sort is faster than more complicated sorting algorithms
Selection sort and insertion sort are “good enough” for small arrays
88
Overview
Divide and Conquer
Merge Sort
Quick Sort
Heap Sort
89
Divide and Conquer1. Base Case, solve the problem
directly if it is small enough
2. Divide the problem into two or more similar and smaller subproblems
3. Recursively solve the subproblems
4. Combine solutions to the subproblems
90Divide and Conquer - Sort
Problem: Input: A[left..right] – unsorted array of
integers Output: A[left..right] – sorted in non-
decreasing order
91
Divide and Conquer - Sort1. Base case
at most one element (left ≥ right), return2. Divide A into two subarrays: FirstPart,
SecondPart Two Subproblems:
sort the FirstPart sort the SecondPart
3. Recursively sort FirstPart sort SecondPart
4. Combine sorted FirstPart and sorted SecondPart
92
Overview
Divide and Conquer
Merge Sort
Quick Sort
Heap Sort
93
Merge Sort: Idea
Merge
Recursively sort
Divide intotwo halves FirstPart SecondPart
FirstPart SecondPart
A
A is sorted!
94Merge Sort: Algorithm
Merge-Sort (A, left, right)
if left ≥ right return
else
middle ← b(left+right)/2
Merge-Sort(A, left, middle)
Merge-Sort(A, middle+1, right)
Merge(A, left, middle, right)
Recursive Call
95
A[middle]A[left]
Sorted FirstPart
Sorted SecondPart
Merge-Sort: Merge
A[right]
merge
A:
A:
Sorted
96
6 10 14 223 5 15 28L: R:
Temporary Arrays
5 15 28 30 6 10 145
Merge-Sort: Merge Example
2 3 7 8 1 4 5 6A:
97Merge-Sort: Merge Example
3 5 15 28 30 6 10 14
L:
A:
3 15 28 30 6 10 14 22R:
i=0
j=0
k=0
2 3 7 8 1 4 5 6
1
98Merge-Sort: Merge Example
1 5 15 28 30 6 10 14
L:
A:
3 5 15 28 6 10 14 22R:
k=1
2 3 7 8 1 4 5 6
2
i=0
j=1
99Merge-Sort: Merge Example
1 2 15 28 30 6 10 14
L:
A:
6 10 14 22R:
i=1
k=2
2 3 7 8 1 4 5 6
3
j=1
100
Merge-Sort: Merge Example
1 2 3 6 10 14
L:
A:
6 10 14 22R:
i=2
j=1
k=3
2 3 7 8 1 4 5 6
4
101
Merge-Sort: Merge Example
1 2 3 4 6 10 14
L:
A:
6 10 14 22R:
j=2
k=4
2 3 7 8 1 4 5 6
i=2
5
102
Merge-Sort: Merge Example
1 2 3 4 5 6 10 14
L:
A:
6 10 14 22R:
i=2
j=3
k=5
2 3 7 8 1 4 5 6
6
103
Merge-Sort: Merge Example
1 2 3 4 5 6 14
L:
A:
6 10 14 22R:
k=6
2 3 7 8 1 4 5 6
7
i=2
j=4
104
Merge-Sort: Merge Example
1 2 3 4 5 6 7 14
L:
A:
3 5 15 28 6 10 14 22R:
2 3 7 8 1 4 5 6
8
i=3
j=4
k=7
105
Merge-Sort: Merge Example
1 2 3 4 5 6 7 8
L:
A:
3 5 15 28 6 10 14 22R:
2 3 7 8 1 4 5 6
i=4
j=4
k=8
106
Merge(A, left, middle, right)1. n1 ← middle – left + 12. n2 ← right – middle3. create array L[n1], R[n2]4. for i ← 0 to n1-1 do L[i] ← A[left +i]5. for j ← 0 to n2-1 do R[j] ← A[middle+j]6. k ← i ← j ← 07. while i < n1 & j < n2 8. if L[i] < R[j] 9. A[k++] ← L[i++]10. else11. A[k++] ← R[j++]12. while i < n1
13. A[k++] ← L[i++]14. while j < n2
15. A[k++] ← R[j++] n = n1+n2
Space: nTime : cn for some constant c
107
6 2 8 4 3 7 5 16 2 8 4 3 7 5 1
Merge-Sort(A, 0, 7)Divide
A:
108
6 2 8 4
3 7 5 1
6 2 8 4
Merge-Sort(A, 0, 3)
, divideA:
Merge-Sort(A, 0, 7)
109
3 7 5 1
8 4
6 26 2
Merge-Sort(A, 0, 1)
, divideA:
Merge-Sort(A, 0, 7)
110
3 7 5 1
8 4
6
2
Merge-Sort(A, 0, 0), base caseA:
Merge-Sort(A, 0, 7)
111
3 7 5 1
8 4
6 2
Merge-Sort(A, 0, 0), returnA:
Merge-Sort(A, 0, 7)
112
3 7 5 1
8 4
6
2
Merge-Sort(A, 1, 1)
, base caseA:
Merge-Sort(A, 0, 7)
113
3 7 5 1
8 4
6 2
Merge-Sort(A, 1, 1), returnA:
Merge-Sort(A, 0, 7)
114
3 7 5 1
8 4
2 6
Merge(A, 0, 0, 1)A:
Merge-Sort(A, 0, 7)
115
3 7 5 1
8 42 6
Merge-Sort(A, 0, 1), returnA:
Merge-Sort(A, 0, 7)
116
3 7 5 1
8 4
2 6
Merge-Sort(A, 2, 3)
48
, divideA:
Merge-Sort(A, 0, 7)
117
3 7 5 1
4
2 6
8
Merge-Sort(A, 2, 2), base caseA:
Merge-Sort(A, 0, 7)
118
3 7 5 1
4
2 6
8
Merge-Sort(A, 2, 2), returnA:
Merge-Sort(A, 0, 7)
119
4
2 6
8
Merge-Sort(A, 3, 3), base caseA:
Merge-Sort(A, 0, 7)
120
3 7 5 1
4
2 6
8
Merge-Sort(A, 3, 3), returnA:
Merge-Sort(A, 0, 7)
121
3 7 5 1
2 6
4 8
Merge(A, 2, 2, 3)A:
Merge-Sort(A, 0, 7)
122
3 7 5 1
2 6 4 8
Merge-Sort(A, 2, 3), returnA:
Merge-Sort(A, 0, 7)
123
3 7 5 1
2 4 6 8
Merge(A, 0, 1, 3)A:
Merge-Sort(A, 0, 7)
124
3 7 5 12 4 6 8Merge-Sort(A, 0, 3), returnA:
Merge-Sort(A, 0, 7)
125
3 7 5 1
2 4 6 8Merge-Sort(A, 4, 7)
A:
Merge-Sort(A, 0, 7)
126
1 3 5 7
2 4 6 8A:Merge (A, 4, 5, 7)
Merge-Sort(A, 0, 7)
127
1 3 5 72 4 6 8Merge-Sort(A, 4, 7), returnA:
Merge-Sort(A, 0, 7)
128
1 2 3 4 5 6 7 8Merge(A, 0, 3, 7)
A:
Merge-Sort(A, 0, 7)Merge-Sort(A, 0, 7), done!
129
Merge-Sort Analysis cn
2 × cn/2 = cn
4 × cn/4 = cn
n/2 × 2c = cn
log n levels
• Total running time: (nlogn)• Total Space: (n)
Total: cn log n
n
n/2 n/2
n/4 n/4 n/4 n/4
2 2 2
130
Merge-Sort Summary
Approach: divide and conquerTime
Most of the work is in the merging Total time: (n log n)
Space: (n), more space than other sorts.
131
Overview
Divide and Conquer
Merge Sort
Quick Sort
Heap Sort
132
Quick Sort
Divide: Pick any element p as the pivot, e.g, the
first element Partition the remaining elements into
FirstPart, which contains all elements < pSecondPart, which contains all elements ≥ p
Recursively sort the FirstPart and SecondPart
Combine: no work is necessary since sorting
is done in place
133
Quick Sort
x < p
p p ≤ x
PartitionFirstPart SecondPart
ppivot
A:
Recursive call
x < p
p p ≤ x
SortedFirstPart
SortedSecondPart
Sorted
134
Quick Sort
Quick-Sort(A, left, right)if left ≥ right return
else middle ← Partition(A, left, right) Quick-Sort(A, left, middle–1 ) Quick-Sort(A, middle+1, right)
end if
135
Partition
p
p x < p
p ≤ x
p p ≤ xx < p
A:
A:
A:p
136
Partition Example
A: 4 8 6 3 5 1 7 2
137
Partition Example
A: 4 8 6 3 5 1 7 2
i=0
j=1
138
Partition Example
A:
j=1
4 8 6 3 5 1 7 2
i=0
8
139
Partition Example
A: 4 8 6 3 5 1 7 26
i=0
j=2
140
Partition Example
A: 4 8 6 3 5 1 7 2
i=0
383
j=3
i=1
141
Partition Example
A: 4 3 6 8 5 1 7 2
i=1
5
j=4
142
Partition Example
A: 4 3 6 8 5 1 7 2
i=1
1
j=5
143
Partition Example
A: 4 3 6 8 5 1 7 2
i=2
1 6
j=5
144
Partition Example
A: 4 3 8 5 7 2
i=2
1 6 7
j=6
145
Partition Example
A: 4 3 8 5 7 2
i=2
1 6 22 8
i=3
j=7
146
Partition Example
A: 4 3 2 6 7 8
i=3
1 5
j=8
147
Partition Example
A: 4 1 6 7 8
i=3
2 542 3
148
A: 3 6 7 81 542
x < 4 4 ≤ x
pivot incorrect position
Partition Example
149
Partition(A, left, right)1. x ← A[left]2. i ← left3. for j ← left+1 to right4. if A[j] < x then 5. i ← i + 16. swap(A[i], A[j])7. end if8. end for j9. swap(A[i], A[left])10. return i
n = right – left +1 Time: cn for some constant c Space: constant
150
4 8 6 3 5 1 7 22 3 1 5 6 7 84
Quick-Sort(A, 0, 7)Partition
A:
151
2 3 1
5 6 7 84
2 1 3
Quick-Sort(A, 0, 7)Quick-Sort(A, 0, 2)
A:
, partition
152
2
5 6 7 84
1
1 3
Quick-Sort(A, 0, 7)Quick-Sort(A, 0, 0)
, base case, return
153
2
5 6 7 84
1
33
Quick-Sort(A, 0, 7)Quick-Sort(A, 1, 1)
, base case
154
5 6 7 842 1 3
2 1 3
Quick-Sort(A, 0, 7)Quick-Sort(A, 2, 2), returnQuick-Sort(A, 0, 2), return
155
42 1 3
5 6 7 86 7 85
Quick-Sort(A, 0, 7)Quick-Sort(A, 2, 2), returnQuick-Sort(A, 4, 7), partition
156
4
5
6 7 87 866
2 1 3
Quick-Sort(A, 0, 7)Quick-Sort(A, 5, 7), partition
157
4
5
6
7 887
2 1 3
Quick-Sort(A, 0, 7)Quick-Sort(A, 6, 7), partition
158
4
5
6
7
2 1 3
Quick-Sort(A, 0, 7)Quick-Sort(A, 7, 7)
8
, return, base case
8
159
4
5
6 87
2 1 3
Quick-Sort(A, 0, 7)Quick-Sort(A, 6, 7)
, return
160
4
5
2 1 3
Quick-Sort(A, 0, 7)Quick-Sort(A, 5, 7)
, return
6 87
161
42 1 3
Quick-Sort(A, 0, 7)Quick-Sort(A, 4, 7)
, return5 6 87
162
42 1 3
Quick-Sort(A, 0, 7)Quick-Sort(A, 0, 7)
, done!5 6 87
163
Quick-Sort: Best Case Even Partition
Total time: (nlogn)
cn
2 × cn/2 = cn
4 × c/4 = cn
n/3 × 3c = cn
log n levels
n
n/2 n/2
n/4
3 3 3
n/4n/4n/4
164
cn
c(n-1)
3c
2c
n
n-1
n-2
3
2
c(n-2)
Happens only if input is sortd input is reversely
sorted
Quick-Sort: Worst Case Unbalanced Partition
Total time: (n2)
165
Quick-Sort: an Average CaseSuppose the split is 1/10 : 9/10
Quick-Sort: an Average Case
cn
cn
cn
≤cn
n
0.1n 0.9n
0.01n 0.09n 0.09n
Total time: (nlogn)
0.81n
2
2
log10n
log10/9n
≤cn
166
Quick-Sort Summary
Time Most of the work done in partitioning. Average case takes (n log(n)) time. Worst case takes (n2) time
Space Sorts in-place, i.e., does not require additional space
167
SummaryDivide and Conquer
Merge-Sort Most of the work done in Merging (n log(n)) time (n) space
Quick-Sort Most of the work done in partitioning Average case takes (n log(n)) time Worst case takes (n2) time (1) space
168
Why study Heapsort?
It is a well-known, traditional sorting algorithm you will be expected to know
Heapsort is always O(n log n)Quicksort is usually O(n log n) but in the
worst case slows to O(n2)Quicksort is generally faster, but Heapsort
is better in time-critical applicationsHeapsort is a really cool algorithm!
169
What is a “heap”?
Definitions of heap:1. A large area of memory from which the
programmer can allocate blocks as needed, and deallocate them (or allow them to be garbage collected) when no longer needed
2. A balanced, left-justified binary tree in which no node has a value greater than the value in its parent
These two definitions have little in common
Heapsort uses the second definition
170
Balanced binary trees
Recall: The depth of a node is its distance from the root The depth of a tree is the depth of the deepest node
A binary tree of depth n is balanced if all the nodes at depths 0 through n-2 have two children
Balanced Balanced Not balanced
n-2n-1n
171
Left-justified binary trees
A balanced binary tree of depth n is left-justified if: it has 2n nodes at depth n (the tree is “full”),
or it has 2k nodes at depth k, for all k < n, and all
the leaves at depth n are as far left as possible
Left-justified Not left-justified
172
Plan of attack
First, we will learn how to turn a binary tree into a heap
Next, we will learn how to turn a binary tree back into a heap after it has been changed in a certain way
Finally (this is the cool part) we will see how to use these ideas to sort an array
173
The heap property
A node has the heap property if the value in the node is as large as or larger than the values in its children
All leaf nodes automatically have the heap property
A binary tree is a heap if all nodes in it have the heap property
12
8 3Blue node has heap property
12
8 12Blue node has heap property
12
8 14Blue node does not have heap property
174
siftUpGiven a node that does not have the heap
property, you can give it the heap property by exchanging its value with the value of the larger child
This is sometimes called sifting upNotice that the child may have lost the heap
property
14
8 12Blue node has heap property
12
8 14Blue node does not have heap property
175
Constructing a heap I
A tree consisting of a single node is automatically a heap
We construct a heap by adding nodes one at a time: Add the node just to the right of the rightmost
node in the deepest level If the deepest level is full, start a new level
Examples:Add a new node here
Add a new node here
176
Constructing a heap II
Each time we add a node, we may destroy the heap property of its parent node
To fix this, we sift upBut each time we sift up, the value of the
topmost node in the sift may increase, and this may destroy the heap property of its parent node
We repeat the sifting up process, moving up in the tree, until either We reach nodes whose values don’t need to be
swapped (because the parent is still larger than both children), or
We reach the root
177
Constructing a heap III
8 8
10
10
8
10
8 5
10
8 5
12
10
12 5
8
12
10 5
8
1 2 3
4
178
Other children are not affected
The node containing 8 is not affected because its parent gets larger, not smaller
The node containing 5 is not affected because its parent gets larger, not smaller
The node containing 8 is still not affected because, although its parent got smaller, its parent is still greater than it was originally
12
10 5
8 14
12
14 5
8 10
14
12 5
8 10
179
A sample heapHere’s a sample binary tree after it has been
heapified
Notice that heapified does not mean sortedHeapifying does not change the shape of the
binary tree; this binary tree is balanced and left-justified because it started out that way
19
1418
22
321
14
119
15
25
1722
180
Removing the root (animated)Notice that the largest number is now in the
rootSuppose we discard the root:
How can we fix the binary tree so it is once again balanced and left-justified?
Solution: remove the rightmost leaf at the deepest level and use it for the new root
19
1418
22
321
14
119
15
1722
11
181
The reHeap method IOur tree is balanced and left-justified, but no longer a heapHowever, only the root lacks the heap property
We can siftUp() the rootAfter doing this, one and only one of its
children may have lost the heap property
19
1418
22
321
14
9
15
1722
11
182
The reHeap method IINow the left child of the root (still the number
11) lacks the heap property
We can siftUp() this nodeAfter doing this, one and only one of its
children may have lost the heap property
19
1418
22
321
14
9
15
1711
22
183
The reHeap method IIINow the right child of the left child of the root
(still the number 11) lacks the heap property:
We can siftUp() this nodeAfter doing this, one and only one of its children may
have lost the heap property —but it doesn’t, because it’s a leaf
19
1418
11
321
14
9
15
1722
22
184
The reHeap method IVOur tree is once again a heap, because every
node in it has the heap property
Once again, the largest (or a largest) value is in the rootWe can repeat this process until the tree becomes emptyThis produces a sequence of values in order largest to smallest
19
1418
21
311
14
9
15
1722
22
185
Sorting
What do heaps have to do with sorting an array?
Here’s the neat part: Because the binary tree is balanced and left justified,
it can be represented as an array Danger Will Robinson: This representation works well
only with balanced, left-justified binary trees All our operations on binary trees can be represented
as operations on arrays To sort:
heapify the array; while the array isn’t empty { remove and replace the root; reheap the new root node;
}
186
Mapping into an array
Notice: The left child of index i is at index 2*i+1 The right child of index i is at index 2*i+2 Example: the children of node 3 (19) are 7 (18) and 8
(14)
19
1418
22
321
14
119
15
25
1722
25 22 17 19 22 14 15 18 14 21 3 9 11 0 1 2 3 4 5 6 7 8 9 10 11 12
187
Removing and replacing the root
The “root” is the first element in the arrayThe “rightmost node at the deepest level” is the last
elementSwap them...
...And pretend that the last element in the array no longer exists—that is, the “last index” is 11 (containing the value 9)
25 22 17 19 22 14 15 18 14 21 3 9 11 0 1 2 3 4 5 6 7 8 9 10 11 12
11 22 17 19 22 14 15 18 14 21 3 9 25 0 1 2 3 4 5 6 7 8 9 10 11 12
188
Reheap and repeatReheap the root node (index 0, containing 11)...
Remember, though, that the “last” array index is changedRepeat until the last becomes first, and the array is sorted!
22 22 17 19 21 14 15 18 14 11 3 9 25 0 1 2 3 4 5 6 7 8 9 10 11 12
9 22 17 19 22 14 15 18 14 21 3 22 25 0 1 2 3 4 5 6 7 8 9 10 11 12
11 22 17 19 22 14 15 18 14 21 3 9 25 0 1 2 3 4 5 6 7 8 9 10 11 12
...And again, remove and replace the root node
189
Analysis I
Here’s how the algorithm starts: heapify the array;
Heapifying the array: we add each of n nodes Each node has to be sifted up, possibly as far as
the root Since the binary tree is perfectly balanced, sifting up
a single node takes O(log n) time Since we do this n times, heapifying takes n*O(log
n) time, that is, O(n log n) time
190
Analysis II
Here’s the rest of the algorithm: while the array isn’t empty { remove and replace the root; reheap the new root node;
}We do the while loop n times (actually, n-1
times), because we remove one of the n nodes each time
Removing and replacing the root takes O(1) time
Therefore, the total time is n times however long it takes the reheap method
191
Analysis III
To reheap the root node, we have to follow one path from the root to a leaf node (and we might stop before we reach a leaf)
The binary tree is perfectly balancedTherefore, this path is O(log n) long
And we only do O(1) operations at each node Therefore, reheaping takes O(log n) times
Since we reheap inside a while loop that we do n times, the total time for the while loop is n*O(log n), or O(n log n)
192
Analysis IV
Here’s the algorithm again: heapify the array; while the array isn’t empty { remove and replace the root; reheap the new root node;
}We have seen that heapifying takes O(n log
n) timeThe while loop takes O(n log n) timeThe total time is therefore O(n log n) + O(n
log n)This is the same as O(n log n) time
193
Homework
1. What is the running time of Merge-Sort if the array is already sorted? What is the best case running time of Merge-Sort?
2. Demonstrate the working of Partition on sequence (13, 5, 14, 11, 16, 12, 1, 15). What is the value of i returned at the completion of Partition?
Searching Algorithms
195
Searching and sorting algorithms with Big O values.
Algorithm Location Big O
Searching Algorithms:
Linear Search Section 24.2.1 O(n)
Binary Search Section 24.2.2 O(log n)
Recursive Linear Search Exercise 24.8 O(n)
Recursive Binary Search Exercise 24.9 O(log n)
Sorting Algorithms:
Selection Sort Section 24.3.1 O(n2)
Insertion Sort Section 24.3.2 O(n2)
Merge Sort Section 24.3.3 O(n log n)
Bubble Sort Exercises 24.5–24.6 O(n2)
196
Number of comparisons for common Big O notations.
n = O(log n) O(n) O(n log n) O(n2)
1 0 1 0 1
2 1 2 2 4
3 1 3 3 9
4 1 4 4 16
5 1 5 5 25
10 1 10 10 100
100 2 100 200 10000
1,000 3 1000 3000 106
1,000,000 6 1000000 6000000 1012
1,000,000,000 9 1000000000 9000000000 1018