Theory of Algorithms:Theory of Algorithms:IntroductionIntroduction

Theory of Algorithms:Theory of Algorithms:IntroductionIntroduction

James Gain and Edwin Blake{jgain | edwin} @cs.uct.ac.za

Department of Computer Science

University of Cape Town

August - October 2004

ObjectivesObjectives

To define an algorithm To introduce:

Problem types The Process of Algorithm Design Solution strategies Ways of Analysing Algorithms

To cover the structure of the course, including practicals

DefinitionsDefinitions

An algorithm is a sequence of unambiguous instructions for solving a problem For obtaining a required output for any legitimate input

in a finite amount of time

Does not require implementation in software

Not an answer but a method for deriving an answer

Historical Perspective: Named after Muhammad ibn Musa al-Khwarizmi – 9th century

mathematician

www.lib.virginia.edu/science/parshall/khwariz.html

Notion of algorithmNotion of algorithm

“computer”

problem

algorithm

input output

• Each step of the algorithm must be unambiguous

• The range of inputs must be specified carefully

• The same algorithm can be represented in different ways

• Several algorithms for solving the same problem may exist - with different properties

What is an algorithm?What is an algorithm?

Recipe, process, method, technique, procedure, routine,… with following requirements:

1. Finiteness Terminates after a finite number of steps

2. Definiteness Rigorously and unambiguously specified

3. Input Valid inputs are clearly specified

4. Output Can be proved to produce the correct output given a valid input

5. Effectiveness Steps are sufficiently simple and basic

Example: SortingExample: Sorting

Statement of problem: Input: A sequence of n numbers <a1, a2, …, an>

Output: A reordering of the input sequence <a´1, a´

2, …, a´

n> so that a´i ≤ a´

j whenever i < j

Instance: The sequence <5, 3, 2, 8, 3>

Algorithms:- Selection sort

- Insertion sort

- Merge sort

- (many others)

Selection SortSelection Sort

Input: array a[1],..,a[n] Output: array a sorted in non-decreasing order Algorithm:

for i=1 to n swap a[i] with smallest of a[i],…,a[n]

for i1 to n domin ifor j i+1 to n do

if a[j] < a[min] min jswap a[i] and a[min]

Exercise: Bridge PuzzleExercise: Bridge Puzzle

Problem: 4 People want to cross a bridge. You have 17 minutes to get them

across

Constraints: It is night and you have 1 flashlight. Max of 2 on the bridge at one

time. All start on the same side

Those crossing must have the flashlight with them. The flashlight must be walked back and forth (no throwing)

People walk at different speeds: person A = 1 minute to cross, person B = 2 minutes, person C = 5 minutes, person D = 10 minutes

A pair walks at the speed of the slower person’s pace

Rumour: this problem is given to Microsoft interviewees

Solution: Bridge PuzzleSolution: Bridge Puzzle

Start (0 min): A B C D

AB Across (2 min): A B C D

A Back (1 min): B A C D

CD Across (10 min): B C D A

B Back (2 min): C D A B

AB Across (2 min): A B C D Total Time = 17 minutes

Extension ExerciseExtension Exercise

This is an instance of a problem. How would you generalise it?

Can you derive an algorithm to solve this generalised problem? Must show the sequence of moves

Must output the minimum time required for crossing

Are there any special cases to watch out for?

Are there any constraints on the input?

Extension SolutionExtension Solution

Input: a list a of crossing times for n people, numbered 1, …, n

Output: total time to cross

Strategy: use 1 & 2 as shuttles and send the others across in pairs

for i 2 to n/2 dot a[2] // 1 & 2 acrosst t + a[1] // 1 backt t + a[i*2] // i*2 & (i*2)-1 acrosst t + a[2] // 2 back

t a[2] // 1 & 2 acrossreturn t

Extension ProblemsExtension Problems

This is an inadequate solution It falsely assumes certain inputs List may not be sorted in ascending order

Sort a

n may not be even numbered Alter final iteration of loop

n > 3 not guaranteed Special case for n = 1, 2, 3

Is not optimal for all inputs, e.g. 1, 20, 21, 22 Can you quantify the nature of these inputs? Suggest an

alternative.

Final solution is left as an exercise. Attempt to make your solution elegant

Fundamentals of Algorithmic Problem Fundamentals of Algorithmic Problem SolvingSolving

Understanding the Problem Make sure you are solving the correct problem and for all legitimate

inputs

Ascertaining the Capabilities of a Computational Device Sequential vs. Parallel.

What are the speed and memory limits?

Choosing between exact and approximate Problem Solving Is absolute precision required? Sometimes this may not be possible

Deciding on Appropriate Data Structures Algorithms often rely on carefully structuring the data

Fundamental Data Structures: array, linked list, stacks, queues, heaps, graphs, trees, sets

Fundamentals of Algorithm DesignFundamentals of Algorithm Design

Applying an Algorithm Design Technique Using a general approach to problem solving that is applicable to a

variety of problems

Specifying the Algorithm Pseudocode is a mixture of natural language and programming

constructs that has replaced flowcharts

Proving an Algorithms Correctness Prove that an algorithm yields a required result for legitimate inputs

in finite time

Analyzing an Algorithm Consider time efficiency, space efficiency, simplicity, generality,

optimality Analysis can be empirical or theoretical

Coding an Algorithm

Well known Computational ProblemsWell known Computational Problems

Sorting Searching String Processing

String Matching

Graph Problems Graph Traversal, Shortest Path, Graph Colouring

Combinatorial Problems Find a combinatorial object - permutation, combination, subset - subject to

constraints

Geometric Problems Closest-Pair, Convex-Hull

Numerical Problems Solving systems of equations, computing definite integrals, evaluating

functions, etc.

Algorithm Design StrategiesAlgorithm Design Strategies

Brute force A straightforward approach to solving a problem, usually

directly based on the problem’s statement

Divide and conquer Divide a problem into smaller instances, solve smaller

instances (perhaps recursively), combine

Decrease and conquer Exploit relationship between the problem and a smaller

instance reduced by some factor (often 1)

Transform and conquer Transform the problem to a simpler instance, another

representation or an instance with a known solution

More Algorithm Design StrategiesMore Algorithm Design Strategies

Greedy approach Make locally optimal steps which (hopefully) lead to a

globally optimal solution for an optimization problem

Dynamic programming Technique for solving problems with overlapping sub-

domains

Backtracking and Branch and bound A way of tackling difficult optimization and combinatorial

problems without exploring all state-space

Space and time tradeoffs Preprocess the input and store additional information to

accelerate solving the problem

How to Solve It: How to Solve It: Understanding the ProblemUnderstanding the Problem

Taken from G. Polya, “How to Solve It”, 2nd edition. A classic textbook on problem solving for mathematics

1. You have to understand the problem. What is the unknown? What are the data? Is the

problem statement sufficient, redundant, contradictory

Draw a figure. Introduce suitable notation

Separate the various parts of the problem. Can you write them down?

Devising a PlanDevising a Plan Find the connection between the data and the unknown. You may be

obliged to consider auxiliary problems if an immediate connection cannot be found. You should obtain eventually a plan of the solution.

Have you seen it before? Or have you seen the same problem in a slightly different form?

Do you know a related problem? Do you know a theorem that could be useful?

Look at the unknown! And try to think of a familiar problem having the same or a similar unknown.

Could you restate the problem? Could you restate it still differently? Go back to definitions.

If you cannot solve the proposed problem try to solve first some related problem. Are the unknown and the new data nearer to each other?

Did you use all the data? Did you use the whole condition? Have you taken into account all essential notions involved in the problem?

Carrying it ThroughCarrying it Through

3. Carry out the Plan Carrying out your plan of the solution, check each

step. Can you see clearly that the step is correct? Can you prove that it is correct?

4. Looking Back Can you check the result? Can you check the

argument?

Can you derive the solution differently? Can you see it at a glance?

Can you use the result, or the method, for some other problem?

Analysis of AlgorithmsAnalysis of Algorithms

How good is the algorithm? Correctness

Time efficiency

Space efficiency

Simplicity

Does there exist a better algorithm? Lower bounds

Optimality

Why Study Algorithms?Why Study Algorithms?

Theoretical importance

The core of computer science

Practical importance

A practitioner’s toolkit of known algorithms

Framework for designing and analyzing algorithms for new problems

Useful mindset

Course StructureCourse Structure

Fundamentals of the Analysis of Algorithms (Ch. 2) Asymptotic notations, analysis of recursive and non-recursive

algorithms, empirical analysis

Algorithmic Strategies (Ch. 3-9) Brute force, Divide-and-Conquer, Decrease-and-Conquer,

Transform-and-Conquer, Space and Time Tradeoffs, Greedy Techniques, Biologically-inspired techniques, Dynamic Programming

Limitations of Algorithms (Ch. 10 + handouts) Turing Machines, Computability, Problem Classification

Coping with Limitations on Algorithms (Ch. 11) Backtracking and Branch and Bound

Anany Levitin, “Introduction to the Design and Analysis of Algorithms”, International Edition, Addison-Wesley, 2003

PracticalsPracticals

Weekly mini prac exams

Given a problem specification that is solvable using the algorithm design strategies presented in the course Design Algorithm

Code it in C++

Submit it for automatic marking

After the 3-hour lab session will be asked to do a short analysis of the solution