Lecture 9CS203

2Math Ahead!

The rest of this lecture uses a few math principles that you learned in HS but may have forgotten.

Do not worry (too much) if your math background is shaky. We introduce mathematical material at a gentle pace. When I started working on my MS here, I had not taken a math class in 25 years, and I managed to learn this material. You can too.

On the other hand, if you want to study this material in more detail, you will not be disappointed. You just have to wait until you take CS312.

3Summations

Summation is the operation of adding a sequence of numbers; the result is their sum or total. Summation is designated with the Greek symbol sigma (∑)

Find the sum

Start from 1

Iterate through values of i

Stop at 100

This summation means "the sum of all integers between 1 and 100 inclusive"

4Useful Mathematic Summations

12

1222....2222

1

1....

2

)1()1(....321

1)1(3210

1)1(3210

nnn

nnn

a

aaaaaaa

nnnn

5Useful Mathematic Summations

The first summation on the previous slide would usually be expressed this way:

For the following values, the value of the summation is 5050, since 100(101)/2 = 10100/2 = 5050:

6Logarithms The logarithm of a number is the exponent to which another

number, the base, must be raised to yield the number. Here is the notation:

logb(y) = x

where b is the base. The parentheses are usually left out in practice. Examples:

log2 8 = 3

log1010,000 = 4

The word "logarithm" is derived (by an early-modern mathematician) from Greek and means roughly "number reasoning." Interestingly (to me, anyway) it is completely unrelated to it anagram "algorithm," which is derived from a Latin version of an Arabic version of the name of the medieval Persian mathematician Kwarizmi

7Logarithms

In other fields, log without an stated base is understood to refer to log10 or loge.

In CS, log without further qualification is understood to refer to log2, pronounced "log base 2" or "binary logarithm." The base is not important in comparing algorithms, but, as you will see, the base is almost always 2 when we are calculating the complexity of programming algorithms.

8Recurrence Relations

A recurrence relation is a rule by which a sequence is generated

Eg, the sequence 5, 8, 11, 14, 17, 20… Is described by the recurrence relation

a0 = 5

an = an-1 + 3

Divide-and-conquer algorithms are often described in terms of recurrence relations

9Analyzing Binary Search Binary Search searches an array or list that is *sorted*

In each step, the algorithm compares the search key value

with the key value of the middle element of the array. If the

keys match, then a matching element has been found and its

index, or position, is returned.

Otherwise, if the search key is less than the middle

element's key, then the algorithm repeats its action on the

sub-array to the left of the middle element or, if the search

key is greater, on the sub-array to the right.

If the remaining array at any step to be searched is empty,

then the key cannot be found in the array and a special "not

found" indication is returned.

10Logarithm: Analyzing Binary Search

Each iteration in binary search contains a fixed number of

operations, denoted by c.

Let T(n) denote the time complexity for a binary search on a list

of n elements. Since we are studying the rate of growth of

execution time, we define T(1) to equal 1.

Assume n is a power of 2; this makes the math simpler and, if it

is not true, the difference is trivial.

Let k=log n. In other words, n = 2k

Since binary search eliminates half of the input after two

comparisons, ncncTckn

Tccn

Tcn

TnTk

log1log)1()2

(...)2

()2

()(2

)(log nO

kn 2

CS-style recurrence relation

11Logarithmic Time

Ignoring constants and smaller terms, the complexity of

the binary search algorithm is O(log n). An algorithm

with the O(log n) time complexity is called a

logarithmic algorithm.

The base of the log is 2, but the base does not affect a

logarithmic growth rate, so it can be omitted.

The time to execute a logarithmic algorithm grows

slowly as the problem size increases. If you square the

input size (with base = 2), the time taken doubles.

12SortingSorting is a classic subject in computer science. There are three reasons for studying sorting algorithms.

First, sorting algorithms illustrate many creative approaches to problem solving that can be applied to other problems.

Second, sorting algorithms are good for practicing fundamental programming techniques using selection statements, loops, methods, and arrays.

Third, sorting algorithms are excellent examples to demonstrate algorithm performance.

13Sorting

These sorting algorithms apply to sorting any type of object, as long as we can find a way to order them. For simplicity, though, we will first sort numeric values, then more complex objects.

When we sort more complex numbers, we will sort them by some key. For example, if class Student has instance variables representing CIN, GPA and name, we will sort according to one of these or some combination of them. We have already done this with the priority queue and other examples.

14Sorting

Arrays and Lists are reference types; that is, the variable we pass between methods does not contain the array or list, but a reference to it.

Therefore, you can sort an array or list in Java with a void method that takes the reference variable, and sorts the elements without returning anything. All other references to the data structure can still be used to access the sorted structure.

On the other hand, you can copy all the elements, construct a new sorted list, and return it. This practice may become more common in the near future for reasons you will learn about in a few weeks.

15Bubble SortRecall that bubble sort repeatedly iterates through the list to be sorted, comparing each pair of adjacent items and swapping them if they are in the wrong order.

This iteration is repeated until no swaps are needed, which indicates that the list is sorted.

The algorithm gets its name from the way smaller elements "bubble" to the top of the list.

Text adapted from Wikipedia

16Bubble Sort

The number of comparisons is always at least as large as the number of swaps. Therefore, in studying the time complexity, we count the comparisons.

The largest key always floats to the right position in the first pass, the next largest rises to the next position in the next pass, etc.

17Bubble Sort

Recall that we are estimating the effect of growth in n, not the exact number of CPU cycles we need. We do not care about the division by 2 since this does not affect the rate of growth.

As n increases, the lower-order term n/2 is dominated by the n2 term. Therefore, we also disregard the lower order term.

Bubble sort time: O(n2)

2212...)2()1(

2 nnnn

In the best case, the data is already sorted, so that we only need one pass, making bubble sort O(n) in this case. More importantly, though, for the worst case, the number of comparisons is :

18Selection Sort and Insertion Sort

These two sorts grow sorted sublists one element at a time.

Selection sort finds the lowest value and moves it to the bottom, or finds the largest element and moves it to the top, then repeats the process for the rest of the values, repeatedly until the list is sorted.

Insertion sort takes one value at a time and places it in the correct spot in the sorted sublist, in the same way most people would sort a hand of cards.

19Analyzing Selection SortThe number of comparisons in selection sort is n-1 for the first iteration, n-2 for the second iteration, and so on. Let T(n) denote the complexity for selection sort and c denote the total number of other operations in each iteration. So,

cnnn

cccncnnT 22

12...)2()1()(2

Ignoring constants and smaller terms, the complexity of

selection sort is O(n2).

20Analyzing Insertion Sort Where selection sort always inserts an element at the end of

the sorted sublist, insertion sort requires inserting some elements in arbitrary places.

At the kth iteration to insert an element to a sorted array of size k, it may take k comparisons to find the insertion position, and, if the structure we are sorting is an array, k moves to insert the element. Let T(n) denote the complexity for insertion sort and c denote the total number of other operations such as assignments in each iteration. So,

Ignoring constants and smaller terms, the complexity of the

insertion sort algorithm is O(n2).

cnnncnccnT 2)1(2...222)(

These two terms are just twice the values for selection sort

21Quadratic Time

An algorithm with O(n2) time complexity is called a

quadratic algorithm. Algorithms with nested loops are often quadratic. A quadratic algorithm's expense grows quickly as

the problem size increases. If you double the input

size, the time for the algorithm is quadrupled.

22Sorting

We often teach bubble sort, selection sort, and insertion sort first because they are easy to understand. Other sort methods are more efficient in average and worst cases.

In particular, there are sort algorithms with O(n log n) complexity. In other words, the consumption of CPU cycles grows proportionally to n times log of n. These algorithms usually involve performing an operation that is O(log n) n times.

Since log n grows much more slowly than n, this is a dramatic improvement over O(n2):

If n = 2, n2 = 4 and n log n = 2 If n = 100, n2 = 10000 and n log n = 664 If n = 10000, n2 = 100,000,000 and n log n = 132,877

23Merge Sort

Merge Sort is a divide and conquer algorithm, like the Towers Of Hanoi algorithm

mergeSort(list): firstHalf = mergeSort(firstHalf); secondHalf = mergeSort(secondHalf); list = merge(firstHalf, secondHalf);

merge:add the lesser of firstHalf [0] and secondHalf [0] to the

new, larger listrepeat until one of the (already sorted) sublists is exhaustedadd the rest of the remaining sublist to the larger list.

24Merge Sort

2 9 5 4 8 1 6 7

2 9 5 4 8 1 6 7

split

2 9

split

5 4

2

split

9 5 4

8 1 6 7

8 1 6 7

2 9

merge

4 5 1 8 6 7

2 4 5 9 1 6 7 8

1 2 4 5 6 7 8 9

merge

merge

divide

conquer

25Merge Two Sorted Lists

2 4 5 9

current1

1

1 6 7 8

current2

current3

(a) After moving 1 to temp (b) After moving all the elements in list2 to temp

to temp

2 4 5 9

current1

1 2 4 5 6 7 8 9

1 6 7 8

current2

current3

(c) After moving 9 to temp

2 4 5 9

current1

1 2 4 5 6 7 8

1 6 7 8

current2

current3

26Merge Sort Time

Assume n is a power of 2. This assumption makes the math simpler. If n is not a power of 2, the difference is trivial.

Merge sort splits the list into two sublists, sorts the sublists using the same algorithm recursively, and then merges the sublists. Each recursive call merge sorts half the list, so the depth of the recursion is the number of times you need to split n to get lists of size 1; this is log n. The single-item lists are, obviously, sorted.

Merge Sort reassembles the list in log n steps, just as it broke the list down.

To merge two subarrays, across all the sublists at one level of recursion, takes at most n-1 comparisons to compare the elements from the two subarrays and n moves to move elements to the new array. The total size of all the sublists is n, the original size of the unsorted list. The total merge time is 2n-1, which is O(n). This happens log n times.

Thus, Merge Sort is O(n log n)

27Merge Sort TimeHere it is again, but with more math.

Let T(n) denote the time required for sorting an array of n elements using merge sort. Without loss of generality, assume n is a power of 2. The merge sort algorithm splits the array into two subarrays, sorts the subarrays using the same algorithm recursively, and then merges the subarrays. So,

The first T(n/2) is the time for sorting the first half of the array and the second T(n/2) is the time for sorting the second half.

mergetimen

Tn

TnT )2

()2

()(

28Merge Sort Time To merge two subarrays takes at most n-1

comparisons to compare the elements from the two subarrays and n moves to move elements to the temporary array. So, the merge time is 2n-1.

)log(1log212log2

1222...22)2

(2

1222...22)2

(2

1222)2

(212)12

2)4

(2(212)2

(2)(

log

1loglog

log

1

22

nnOnnnnn

nnnn

T

nnnn

T

nnn

Tnnn

Tnn

TnT

n

nn

n

kk

k

2log n = n and T(1) = 1

2n – 20

1

1....

1)1(3210

a

aaaaaaa

nnn

a =2 so a-1 = 1

Subtractive term, so the -1 from the summation becomes +1

29Quick SortQuick sort, developed by C. A. R. Hoare (1962), works as follows:

Select an element, called the pivot, in the array. Divide the array into two parts such that all the

elements in the first part are less than or equal to the pivot and all the elements in the second part are greater than the pivot.

Recursively apply the quick sort algorithm to the first part and then the second part.

30

Quick Sortfunction quicksort(a)

// an array of zero or one elements is already sortedif length(a) ≤ 1 return a

select and remove a pivot element pivot from arraycreate empty lists less and greater

for each x in a if x ≤ pivot then append x to less else append x to greater // two recursive calls return concatenate(quicksort(less), list(pivot),

quicksort(greater))

The earliest version of quicksort used the first index as the pivot, and demos of quicksort often still do this for simplicity. However, in an already-sorted array, this will cause the worst case O(n2) behavior. The middle index is safer, although yet more complex solutions to this problem also exist.

31Quick Sort

5 2 9 3 8 4 0 1 6 7

pivot

(a) The original array

4 2 1 3 0 5 8 9 6 7

pivot

(b)The original array is partitioned

0 2 1 3 4 (c) The partial array (4 2 1 3 0) is

partitioned

0 2 1 3 (d) The partial array (0 2 1 3) is partitioned

1 2 3

pivot

pivot

pivot

(e) The partial array (2 1 3) is partitioned

32

Quick Sort

How to accomplish the partition in a single array:

Search from the beginning of the list for the first element greater than the pivot and from the end for the last element less than the pivot.

When found, swap them. Continue until the list is partitioned.

Finally, swap the last element in the left partition with the pivot

33

package demos;

import javax.swing.JOptionPane;

public class Demo {

// http://www.mycstutorials.com/articles/sorting/quicksortpublic void quickSort(int array[]) {

for (int x : array) System.out.print(x + " ");System.out.println();quickSort(array, 0, array.length - 1);

}

public void quickSort(int array[], int start, int end) {int i = start; // index of left-to-right scanint k = end; // index of right-to-left scanif (end - start >= 1) // check that there are at least two elements{

int pivot = array[start]; // set the first element as pivot

System.out.println("pivot " + pivot);System.out.println("i: " + i + " k: " + k);while (k > i) // while the scan indices have not met{

while (array[i] <= pivot && i <= end && k > i) {// from the left, look for the firsti++; // element greater than the pivotSystem.out.println("i: " + i);

}while (array[k] > pivot && k >= start && k >= i) {

// from the right, look for the firstk--; // element not greater than the pivotSystem.out.println("k: " + k);

}if (k > i) // if the left seekindex is still smaller than

swap(array, i, k); // the right index, swap the corresponding elements

}swap(array, start, k); // after the indices cross, swap the last element in the left partition with the

// pivotquickSort(array, start, k - 1); // quicksort the left partitionquickSort(array, k + 1, end); // quicksort the right partition

}

34

else // if there is only one element in the partition, do not do any sorting

{return; // the array is sorted, so exit

}}

public void swap(int array[], int index1, int index2){

int temp = array[index1]; // store the first value in a temparray[index1] = array[index2]; // copy the value of the second into the

// firstarray[index2] = temp; // copy the value of the temp into the secondfor (int x : array)

System.out.print(x + " ");System.out.println();

}

public static void main(String[] args) {Demo q = new Demo();int[] myArray = { 5, 4, 10, 11, 9, 8, 1 };q.quickSort(myArray);

}}

35

5 2 9 3 8 4 0 1 6 7

pivot low high

(a) Initialize pivot, low, and high

5 2 9 3 8 4 0 1 6 7

pivot low high

(b) Search forward and backward

5 2 1 3 8 4 0 9 6 7

pivot low high

(c) 9 is swapped with 1

5 2 1 3 8 4 0 9 6 7

pivot low high

(d) Continue search

5 2 1 3 0 4 8 9 6 7

pivot low high

(e) 8 is swapped with 0

5 2 1 3 0 4 8 9 6 7

pivot low high

(f) when high < low, search is over

4 2 1 3 0 5 8 9 6 7

pivot

(g) pivot is in the right place

The index of the pivot is returned

36Quick Sort Partition Time

To partition an array of n elements takes n-1 comparisons and n moves in the worst case. So, the time required for partition is O(n).

37Worst-Case Time

In the worst case, each time the pivot divides the array into one big subarray with the other empty.

The size of the big subarray is one less than the one before divided, so the O(n) partitioning occurs n-1 times.

Worst-case time:

)(12...)2()1( 2nOnn

38Best-Case Time

In the best case, each time the pivot divides the array into two parts of about the same size, so we partition log n times.

Since the O(n) partitioning occurs log n times,Quicksort is O(n log n) in this case.

39Average-Case Time

On the average, each time the pivot will not divide the array into two parts of the same size nor one empty part.

Statistically, the sizes of the two parts are very close. So the average time is O(n logn). The exact eprformance depends on the data.

40Bucket Sort All sort algorithms discussed so far are general sorting

algorithms that work for any types of keys (e.g., integers, strings, and any comparable objects).

These algorithms sort the elements by comparing their keys. The lower bound for general sorting algorithms is O(nlogn). So, no sorting algorithms based on comparisons can perform better than O(n log n).

However, if the keys are small integers, you can use bucket sort without having to compare the keys.

41Bucket SortThe bucket sort algorithm works as follows. Assume the keys are in the range from 0 to N-1. We need N buckets labeled 0, 1, ..., and N-1. If an element’s key is i, the element is put into the bucket i. Each bucket holds the elements with the same key value. You can use an ArrayList to implement a bucket.

Bucket Sort is O(n)

bucket[0]

Elements with key 0

bucket[1]

Elements with key 1

bucket[2]

Elements with key 2

…

bucket[N-1]

Elements with key N-1

42Common Recurrence Relations Recurrence Relation Result Example

)1()2/()( OnTnT )(log)( nOnT Binary search, Euclid’s GCD

)1()1()( OnTnT )()( nOnT Linear search

)1()2/(2)( OnTnT )()( nOnT

)()2/(2)( nOnTnT )log()( nnOnT Merge sort (Chapter 24)

)log()2/(2)( nnOnTnT )log()( 2 nnOnT

)()1()( nOnTnT )()( 2nOnT Selection sort, insertion sort

)1()1(2)( OnTnT )2()( nOnT Towers of Hanoi

)1()2()1()( OnTnTnT )2()( nOnT Recursive Fibonacci algorithm

43Comparing Common Growth Functions

Constant time

)2()()()log()()(log)1( 32 nOnOnOnnOnOnOO

)1(O

)(log nO Logarithmic time )(nO Linear time

)log( nnO Log-linear time )( 2nO Quadratic time

)( 3nO Cubic time )2( nO Exponential time

44Comparing Common Growth Functions

)2()()()log()()(log)1( 32 nOnOnOnnOnOnOO

O(1)

O(logn)

O(n)

O(nlogn)

O(n2) O(2n)

.jar files.jar files are used for distributing Java applications and libraries.

The file format is .zip, but the extension .jar identifies them as Java Archives

They contain bytecode (.class files), any other files from the application or library (like images or audio files), and can also contain source code

The JDK contains command line tools for making jar files, but this is easier to do with Eclipse

Jar files may be executable, meaning that they are configured to launch the main() of some class contained in the jar

.jar files

.jar files