1.2. Comparing Algorithms

Learning outcomes

Understand that algorithms can be compared by expressing their complexity as a function relative to the size of the problem.

Understand that some algorithms are more efficient time-wise than other algorithms.

Understand that some algorithms are more space-efficient than other algorithms.

Understand the Order of Complexity:Linear time, polynomial time, exponential time.

What is an algorithm?

Sequence of unambiguous instructions for solving a problem.

For obtaining an output for any legitimate input in a finite amount of time.=Turing Machine program

Why comparing algorithms?

It is not good enough to just solve a problem; a problem needs to be solved efficiently

Two measures:TimeMemory

Only becomes important with large amount of data or large problems

Complexity (space or time)

Look at how time (or space) grows as size of problem input grows.=> Big O Notation

Complexity is measured irrespectively of the machine speed

Many programmers have had ugly surprises when they moved from small test data to large data sets. This analysis will make you aware of potential problems

Count 0

For Count 1 to n

Sum Sum + Count

End for

Output Sum

Natural Number 1 2 3 4 5 6

Running Total 3 6 10 1521

n=10

n=1

∑ n=55n=10

n=1

∑ n=55

Mathematical Series

Sum n = ½ n (n+1)

Pseudocode :Sum n * (n + 1)/2 //compute sum of numbers

Output Sum

Sum=1/2 * 10 * (10+1)

= 55 (reassuringly!)

∑

Activity Code both summation algorithms in VB.net

and run to prove the answer is the same Declare the variable n as double and use

VERY large numbers (100,000,000) and see if you can output the elapsed time for both methods.

Count = 0For Count = 1 to n

Sum = Sum + CountEnd forOutput Sum

Sum = n * (n + 1)/2

Summary

Always: an algorithm must be correct for ALL valid inputs!!

But: the amount of memory used and speed of computation for a given set of values are also important criteria.

Time efficiency depends on input size!

Order of Complexity

Big-O (the "O" stands for "order of") notation is concerned with what happens for very large values of N, therefore only the largest term in a polynomial is needed. All smaller terms are dropped.

Big-O describes how long it takes for an algorithm to be executed.

For example, the number of operations (comparisons) in some sorts is N2 - N. (Example?)

For large values of N, the single N term is insignificant compared to N2, therefore one of these sorts would be described as an O(N2) algorithm.

Here is a table of typical cases, showing how many "operations" would be performed for various values of N. Logarithms to base 2 (as used here) are proportional to logarithms in other base, so this doesn't affect the big-O formula.

  constant logarithmic linear   quadratic cubic

n O(1) O(log N) O(N) O(N log N) O(N2) O(N3)

1 1 1 1 1 1 1

2 1 1 2 2 4 8

4 1 2 4 8 16 64

8 1 3 8 24 64 512

16 1 4 16 64 256 4,096

1,024 1 10 1,024 10,2401,048,57

6

1,073,741,82

4

1,048,576 1 201,048,

57620,971,520 1012 1016

Does anyone really have that much data?

It's quite common. For example, it's hard to find a digital camera

that has fewer than a million pixels (1 mega-pixel). These images are processed and displayed on the screen.

The algorithms that do this had better not be O(N2)! If it took one microsecond (1 millionth of a second) to process each pixel, an O(N2) algorithm would take more than a week to finish processing a 1 megapixel image, and more than three months to process a 3 megapixel image (note the rate of increase is definitely not linear).

Another example is sound. CD audio

samples are 16 bits, sampled 44,100 times per second for each of two channels. A typical 3 minute song consists of about 8 million data points. You had better choose the right algorithm to process this data.

A dictionary used for text analysis has about 125,000 entries. There's a big difference between a linear O(N), binary O(log N), or hash O(1) search.

Searching

Type of Search Big-Oh Comments

Linear search array/ArrayList/LinkedList

O(N)  

Binary search sorted array/ArrayList

O(log N) Requires sorted data.

Search balanced tree O(log N)  

Search hash table O(1)  

Sorting arrays/ArrayLists Type of

SortBest Worst Average Comments

BubbleSort O(N) O(N2) O(N2) Not a good sort, except with ideal data.

Insertion sort

O(N) O(N2) O(N2) Perhaps best of O(N2) sorts

QuickSort O(N log N) O(N2) O(N log N) Good, but it worst case is O(N2)

Overview

Constant complexity O(1) – takes the same amount of time no matter how big the problem is.

Linear complexity O(n) – as the size of the problem increases, the time taken to solve the problem grows at the same rate.

Logarithmic complexity O(log n) – as the size of the problem grows, the time taken to solve the problem increases, but at a slower rate.

Polynomial complexity O(nk) – as the size of the problem grows, the time taken to solve it grows faster, but can be solved in a reasonable amount of time.

Exponential complexity O(kn) – as the size of the problem grows, the time taken to solve it grows by larger and larger amounts. Only small size versions of these problems can be solved in a reasonable amount of time.

Factorial complexity O(n!) – even worse than exponential.