Recursion. Recursion: Definition Function that solves a problem by relying on itself to compute the...

RecursionRecursion

Recursion: DefinitionRecursion: Definition

Function that solves a problem by Function that solves a problem by relying on itself to compute the relying on itself to compute the correct solution for a correct solution for a smallersmaller version version of the problemof the problem

Requires Requires terminating conditionterminating condition: : Case for which recursion is no Case for which recursion is no longer neededlonger needed

Factorial RecursionFactorial Recursion

Factorial:Factorial: n! = n * (n-1)!n! = n * (n-1)! Base Case => 0! = 1Base Case => 0! = 1 Smaller problem => Solving (n-1)!Smaller problem => Solving (n-1)!

Implementation:Implementation:

long factorial(long inputValue)long factorial(long inputValue)

{{

if (inputValue == 0) return 1;if (inputValue == 0) return 1;

else return inputValue * else return inputValue * factorial(inputValue - 1);factorial(inputValue - 1);

}}

SearchingSearching We want to find whether or not an input value is We want to find whether or not an input value is

in a sorted list:in a sorted list:

8 in [1, 2, 8, 10, 15, 32, 63, 64]?8 in [1, 2, 8, 10, 15, 32, 63, 64]?33 in [1, 2, 8, 10, 15, 32, 63, 64]?33 in [1, 2, 8, 10, 15, 32, 63, 64]?------------------------------------------------------------------------------------------------------------

int index = 0;int index = 0;while (index < listSize)while (index < listSize){{

if (list[index] == input) return index;if (list[index] == input) return index;index++;index++;

}}return –1;return –1;

SearchingSearching

Better method? Better method? Use fact that we know the list is sortedUse fact that we know the list is sorted Cut what we have to search in half each timeCut what we have to search in half each time

Compare input to middleCompare input to middle If input greater than middle, our value has be in If input greater than middle, our value has be in

the elements on the right side of the middle the elements on the right side of the middle elementelement

If input less than middle, our value has to be in the If input less than middle, our value has to be in the elements on the left side of the middle elementelements on the left side of the middle element

If input equals middle, we found the element.If input equals middle, we found the element.

Searching: Binary SearchSearching: Binary Search

for (int left = 0, right = n –1; left <= right;)for (int left = 0, right = n –1; left <= right;)

{{

middle =(left + right) / 2;middle =(left + right) / 2;

if (input == list[middle]) return middle;if (input == list[middle]) return middle;

else if (input < list[middle]) right = middle else if (input < list[middle]) right = middle – 1;– 1;

else left = middle + 1;else left = middle + 1;

}}

return – 1;return – 1;


8 in [1, 2, 8, 10, 15, 32, 63, 64]?8 in [1, 2, 8, 10, 15, 32, 63, 64]?11stst iteration: iteration:

Left = 0, Right = 7, Middle = 3, List[Middle] = Left = 0, Right = 7, Middle = 3, List[Middle] = 1010

Check 8 == 10 => No, 8 < 10Check 8 == 10 => No, 8 < 10

22ndnd iteration: iteration:Left = 0, Right = 2, Middle = 1, List[Middle] = 2Left = 0, Right = 2, Middle = 1, List[Middle] = 2Check 8 == 2 => No, 8 > 2Check 8 == 2 => No, 8 > 2

33rdrd iteration: iteration:Left = 2, Right = 2, Middle = 2Left = 2, Right = 2, Middle = 2Check 8 == 8 => Yes, Found It!Check 8 == 8 => Yes, Found It!


Binary Search Method:Binary Search Method: Number of operations to find input if in Number of operations to find input if in

the list:the list: Dependent on position in listDependent on position in list 1 operation if middle1 operation if middle LogLog22 n operations maximum n operations maximum

Number of operations to find input if Number of operations to find input if not in the list:not in the list: LogLog22 n operations maximum n operations maximum

Recursive Binary SearchRecursive Binary Search

Two requirements for recursion:Two requirements for recursion: Same algorithm, smaller problemSame algorithm, smaller problem Termination conditionTermination condition

Binary search?Binary search? Search in half of previous arraySearch in half of previous array Stop when down to one elementStop when down to one element

Recursive Binary SearchRecursive Binary Searchint BinarySearch(int *list, const int input, const int left, int BinarySearch(int *list, const int input, const int left,

const int right)const int right){{

if (left < right)if (left < right){{

middle =(left + right) / 2;middle =(left + right) / 2;if (input == list[middle]) return middle;if (input == list[middle]) return middle;else if (input < list[middle]) return else if (input < list[middle]) return

BinarySearch(list, input, left, middle-1);BinarySearch(list, input, left, middle-1);else return else return

BinarySearch(list,input,middle+1,right);BinarySearch(list,input,middle+1,right);}}

return – 1;return – 1;}}

Fibonacci ComputationFibonacci Computation

Fibonacci Sequence:Fibonacci Sequence: 1, 1, 2, 3, 5, 8, 13, 21, 34, 55, 89, …1, 1, 2, 3, 5, 8, 13, 21, 34, 55, 89, … Simple definition:Simple definition: Fib[0] = 1Fib[0] = 1 Fib[1] = 1Fib[1] = 1 Fib[N] = Fib(N-1) + Fib(N-2)Fib[N] = Fib(N-1) + Fib(N-2)

Recursive FibonacciRecursive Fibonacci

int fibonacci(int input)int fibonacci(int input)

{{

if ((input == 0) || (input == 1)) if ((input == 0) || (input == 1)) return 1;return 1;

else return (fibonacci(input-1) + else return (fibonacci(input-1) + fibonacci(input-2));fibonacci(input-2));

}}

Iterative FibonacciIterative Fibonacciint fibonacci(int input)int fibonacci(int input){{ int first = 1;int first = 1; int second = 1;int second = 1;

int temp;int temp; for (int k = 0; k < input; k++)for (int k = 0; k < input; k++) {{ temp = first;temp = first; first = second;first = second; second = temp + second;second = temp + second; }} return first;return first;}}

Types of RecursionTypes of Recursion

Linear Recursion:Linear Recursion: 1 recursive call per function1 recursive call per function Factorial, Binary Search examplesFactorial, Binary Search examples

Tree Recursion:Tree Recursion: 2 or more recursive calls per function2 or more recursive calls per function Fibonacci ExampleFibonacci Example

Efficiency of RecursionEfficiency of Recursion

Recursion can sometimes be slower Recursion can sometimes be slower than iterative codethan iterative code

Two main reasons:Two main reasons: Program stack usageProgram stack usage

When using recursive functions, every When using recursive functions, every recursive call is added to the stack and it recursive call is added to the stack and it grows fast.grows fast.

Result generationResult generation

Efficiency of RecursionEfficiency of Recursion Stack Usage:Stack Usage:

When a function is called by a program, that function is When a function is called by a program, that function is placed on the program call stack:placed on the program call stack:

Every stack entry maintains information about the Every stack entry maintains information about the function:function: Where to return to when the function completesWhere to return to when the function completes Storage for local variablesStorage for local variables Pointers or copies of arguments passed inPointers or copies of arguments passed in

main()

getData()

readFile()Returns file data to be used in getData()

Returns formatted data to be printed in main()

Efficiency of RecursionEfficiency of Recursion Another Reason for Slowdowns [Tree Recursion]Another Reason for Slowdowns [Tree Recursion]

Traditional Recursion doesn’t save answers as it executesTraditional Recursion doesn’t save answers as it executes Fib(5) Fib(5) = Fib(4) + Fib(3)= Fib(4) + Fib(3)= Fib(3) + Fib(2) + Fib(3)= Fib(3) + Fib(2) + Fib(3)= Fib(2) + Fib(1) + Fib(2) + Fib(3)= Fib(2) + Fib(1) + Fib(2) + Fib(3)= Fib(1) + Fib(0) + Fib(1) + Fib(2) + Fib(3)= Fib(1) + Fib(0) + Fib(1) + Fib(2) + Fib(3)= Fib(1) + Fib(0) + Fib(1) + Fib(1) + Fib(0) + Fib(3)= Fib(1) + Fib(0) + Fib(1) + Fib(1) + Fib(0) + Fib(3)= Fib(1) + Fib(0) + Fib(1) + Fib(1) + Fib(0) + Fib(2) + = Fib(1) + Fib(0) + Fib(1) + Fib(1) + Fib(0) + Fib(2) +

Fib(1)Fib(1)= Fib(1) + Fib(0) + Fib(1) + Fib(1) + Fib(0) + Fib(1) + = Fib(1) + Fib(0) + Fib(1) + Fib(1) + Fib(0) + Fib(1) +

Fib(0) + Fib(1)Fib(0) + Fib(1) Solution: Dynamic programming – saving answers Solution: Dynamic programming – saving answers

as you go and reusing themas you go and reusing them

Dynamic ProgrammingDynamic Programming

Fibonacci ProblemFibonacci Problem We know the upper bound we are solving forWe know the upper bound we are solving for I.e. Fibonacci (60) = 60 different answersI.e. Fibonacci (60) = 60 different answers Generate an array 60 long and initialize to –Generate an array 60 long and initialize to –

11 Everytime we find a solution, fill it in in the Everytime we find a solution, fill it in in the

arrayarray Next time we look for a solution, if the value Next time we look for a solution, if the value

in the array for the factorial we need is not –in the array for the factorial we need is not –1, use the value present.1, use the value present.

Algorithmic Algorithmic ComplexityComplexity

Algorithmic Complexity:Algorithmic Complexity:Two Key MetricsTwo Key Metrics

Space Efficiency:Space Efficiency: The The maximummaximum amount of memory required to amount of memory required to

perform a computation and how that maximum perform a computation and how that maximum amount is related to the amount is related to the size of the inputsize of the input

Time Efficiency:Time Efficiency: The The maximummaximum number of computations required number of computations required

to perform a computation and how that maximum to perform a computation and how that maximum amount is related to the amount is related to the size of the inputsize of the input

We’ll look at these in terms of recursion first, We’ll look at these in terms of recursion first, then extend to more general discussionthen extend to more general discussion..

Determining Time Determining Time EfficiencyEfficiency

Two ways to model:Two ways to model: Experimental: Adding in counters to measure Experimental: Adding in counters to measure

the number of operations performedthe number of operations performed

Theoretical: Using mathematical model of a Theoretical: Using mathematical model of a functions computational requirementsfunctions computational requirements

Recursion? Some problems have very nice Recursion? Some problems have very nice mathematical functions - Let’s start with mathematical functions - Let’s start with those first.those first.

Recursion Time Efficiency: Recursion Time Efficiency: Recurrence RelationsRecurrence Relations

Computational Requirements: Three Computational Requirements: Three IssuesIssues Amount of work needed for current iterationAmount of work needed for current iteration

Cost of basic function requirementsCost of basic function requirements Number of subproblems that have to be Number of subproblems that have to be

solvedsolved Linear recurrence vs Tree recurrenceLinear recurrence vs Tree recurrence

Size (in terms of input) of subproblems to be Size (in terms of input) of subproblems to be solvedsolved

How much smaller is each of the subproblemsHow much smaller is each of the subproblems


Recursive Binary Search Requirements:Recursive Binary Search Requirements: Amount of work needed for current iterationAmount of work needed for current iteration

1 Comparison (inputValue versus middle index)1 Comparison (inputValue versus middle index) 1 Adjustment (Changing left or right parameters)1 Adjustment (Changing left or right parameters)

Number of subproblems that have to be solvedNumber of subproblems that have to be solved 1 Subproblem (the left or right side of the array)1 Subproblem (the left or right side of the array)

Size (in terms of input) of subproblems to be solvedSize (in terms of input) of subproblems to be solved Subproblem is half the size of the original problemSubproblem is half the size of the original problem

Recurrence RelationRecurrence Relation

General Recurrence Relation:General Recurrence Relation:

T(n) = aT(n/b) + cnT(n) = aT(n/b) + cnkk

a = number of sub a = number of sub problemsproblems

b = 1/size of subproblemsb = 1/size of subproblemsf(n) = current iteration f(n) = current iteration

work = work = constant * nconstant * nkk

Recurrence: Master Recurrence: Master TheoremTheorem

T(n) = aT(n/b) + f (n) where f T(n) = aT(n/b) + f (n) where f (n) (n) ≈≈ n nkk

1.1. a < ba < bkk T(n) ~ n T(n) ~ nkk

2.2. a = ba = bkk T(n) ~ n T(n) ~ nk k lg n lg n

3.3. a > ba > bkk T(n) ~ T(n) ~ nnlog log b b aa

Recurrence: Master Recurrence: Master TheoremTheorem

T1(n) = 7 T1(n/7)+n a = 7, b = 7, k = 1 a ? bk 7 == 7 nklgn=> n lgn

T2(n) = 7 T2(n/2)+n2

a = 7, b = 2, k = 2 a ? bk 7 > 22

=> nlogb

a => nlog2

7 => n2.81

T3(n) = 7 T3(n/3)+n2

a = 7, b = 3, k = 2 a ? bk 7 < 32

=> nk => n2


T(n) = aT(n/b) + f(n)T(n) = aT(n/b) + f(n)a = number of sub problemsa = number of sub problems = 1= 1b = 1/size of subproblemsb = 1/size of subproblems = 1/(1/2) => 2= 1/(1/2) => 2f(n) = current iteration workf(n) = current iteration work = 2n= 2n00 so k = 0 so k = 0

Compare a to bCompare a to bkk:: 1 vs 2^01 vs 2^0 = 1 vs 1= 1 vs 1If they are equal, computational cost is:If they are equal, computational cost is:

nnkk log n log n = 1 * log n= 1 * log n =>=> log log nn

[Formula can be looked up for >, <, and ==][Formula can be looked up for >, <, and ==]

Space ComplexitySpace Complexity

Why are we interested in space Why are we interested in space complexity?complexity? Most people worry about time complexity. Most people worry about time complexity.

In general, problems with high time complexity In general, problems with high time complexity can still be solved – we just have to wait longer can still be solved – we just have to wait longer (albeit there are some problems that take so long (albeit there are some problems that take so long they are not solvable in our lifetime/multiple they are not solvable in our lifetime/multiple lifetimes).lifetimes).

Problems with too large space complexity we Problems with too large space complexity we may not be able to run at all. Our finiteness may not be able to run at all. Our finiteness of memory is more much more evident than of memory is more much more evident than our finiteness of time.our finiteness of time.


In general, two components:In general, two components: Fixed requirementsFixed requirements CC

Instruction space (code)Instruction space (code) ConstantsConstants Simple variablesSimple variables

Instance requirementsInstance requirements SSpp

Variables whose size is dependent on problem Variables whose size is dependent on problem instanceinstance

Recursion stack spaceRecursion stack space

S(P) = C + SS(P) = C + Spp

Ignore C as it is usually dominated by SIgnore C as it is usually dominated by Spp


float abc(float a, float b, float c)float abc(float a, float b, float c){{

return a + b + b*c + (a+b-c) / (a+b) + 4.0;return a + b + b*c + (a+b-c) / (a+b) + 4.0;}}

Assume space required is 1 word for each Assume space required is 1 word for each float.float.

No matter what a, b, c are entered, need No matter what a, b, c are entered, need same amount of memory to execute.same amount of memory to execute.

Space required Space required for this functionfor this function is is independent of inputsindependent of inputs, so S, so Sabcabc = 0 = 0


float sum(float *a, const int n)float sum(float *a, const int n){{

float s = 0;float s = 0;for (int i =0; i < n; i++) s += a[i];for (int i =0; i < n; i++) s += a[i];return s;return s;

}}

Summing over an entire array, problem size = Summing over an entire array, problem size = array length [n]. Need four words - one word array length [n]. Need four words - one word apiece for a, n, s, i. Again requirements apiece for a, n, s, i. Again requirements for for this functionthis function are are independent of inputsindependent of inputs, so , so SSsumsum = 0 = 0 Note the difference between this function space Note the difference between this function space

requirements and program as a whole (somebody has requirements and program as a whole (somebody has to allocate the four words for the data in the array).to allocate the four words for the data in the array).


float rsum(float *a, const int n)float rsum(float *a, const int n){{

if (n <= 0) return 0;if (n <= 0) return 0;else return (rsum(a, n-1) + a[n-1]);else return (rsum(a, n-1) + a[n-1]);

}}

Summing over an entire array, problem size = Summing over an entire array, problem size = array length [n]. Need four words - one array length [n]. Need four words - one word apiece for a, n, s, i. word apiece for a, n, s, i.

Every recursive call needs those four words, Every recursive call needs those four words, and depth of recursion is n + 1 (base = 0, and depth of recursion is n + 1 (base = 0, 1..n)1..n)

Requirements Requirements for this functionfor this function are are dependent on inputsdependent on inputs, so S, so Srsumrsum = 4(n+1) = 4(n+1)

Time ComplexityTime Complexity

T(P)T(P) = Compile time + Execution Time= Compile time + Execution Time

Compile time is:Compile time is: Not dependent on program inputsNot dependent on program inputs Negligible in comparison to execution time Negligible in comparison to execution time

for real world, extensively used programsfor real world, extensively used programs

Execution time:Execution time: Difficult to accurately obtain Difficult to accurately obtain

Hardware dependent, OS dependent, load Hardware dependent, OS dependent, load dependent …dependent …

Estimate in terms of program stepsEstimate in terms of program steps

Time Complexity in C++ Time Complexity in C++ ContextContext

Comments: Comments: Not executedNot executed Step count = 0Step count = 0

Declarations: (int b; )Declarations: (int b; ) Either not executable or lumped into Either not executable or lumped into

cost of calling functioncost of calling function Step count = 0Step count = 0

C++ Time ComplexityC++ Time Complexity Expressions: (for example: a == true) Expressions: (for example: a == true)

Generally step count = 1Generally step count = 1 If expression contains functions, sum of costs If expression contains functions, sum of costs

for function invocations (defined later)for function invocations (defined later) Assignments: <variable> = <expression>Assignments: <variable> = <expression>

Generally, step count for evaluation of Generally, step count for evaluation of expressionexpression

If variable size dependent on instance,If variable size dependent on instance, step count = size of variable + expression step countstep count = size of variable + expression step count

I.E. a = (2+b); a,b both lists, then: I.E. a = (2+b); a,b both lists, then: step count = step count = cost of evaluating expression (2+b) + cost of evaluating expression (2+b) +

cost of copying list b into list acost of copying list b into list a

C++ Time ComplexityC++ Time Complexity Iteration: Iteration:

Consider loop control elements, not internal Consider loop control elements, not internal statements of the loop bodystatements of the loop body

While <expr> doWhile <expr> do Do… while <expr>Do… while <expr> For (<init_stmt>; <expr1>; <expr2>)For (<init_stmt>; <expr1>; <expr2>)

while, dowhile, do: Each iteration requires step count of : Each iteration requires step count of <expr><expr>

forfor: : Generally 1 for each iteration unless <init_stmt>, Generally 1 for each iteration unless <init_stmt>,

<expr1>, <expr2> dependent on size<expr1>, <expr2> dependent on size 11stst time (initialize, test) time (initialize, test) = <init_stmt> steps + <expr1> steps = <init_stmt> steps + <expr1> steps All other times (update, test)All other times (update, test) = <expr1> steps + <expr2> = <expr1> steps + <expr2>

steps steps Generally have to count iterations + 1 (all true iterations Generally have to count iterations + 1 (all true iterations

where fall into body plus one where the condition fails) where fall into body plus one where the condition fails)

C++ Time ComplexityC++ Time Complexity Switch statement: Switch statement:

switch <expr> { case cond1: <statement1> switch <expr> { case cond1: <statement1> case cond2: <statement2> default: case cond2: <statement2> default: <statement>}<statement>}

HeaderHeader: <expr> steps: <expr> steps ConditionsConditions: Cost of own check plus cost of all : Cost of own check plus cost of all

preceding checks preceding checks If/else: If/else:

If (<expr>) <statements1>; else If (<expr>) <statements1>; else <statements2><statements2>

<expr> = true<expr> = true: <expr> +<statements1> cost: <expr> +<statements1> cost <expr> = false<expr> = false: <expr>+ <statements2> cost: <expr>+ <statements2> cost

C++ Time ComplexityC++ Time Complexity

Function invocation:Function invocation: Step cost of 1 generallyStep cost of 1 generally Pass by value parameters: If variable Pass by value parameters: If variable

size dependent on problem instance, size dependent on problem instance, add step cost of size (to do copy)add step cost of size (to do copy)

If recursive, include all copies of local If recursive, include all copies of local variables that are size dependent variables that are size dependent because must be generated in recursive because must be generated in recursive callcall

C++ Time ComplexityC++ Time Complexity

Memory managementMemory management: new/delete: new/delete Step cost 1 generallyStep cost 1 generally If call constructor, destructor, compute If call constructor, destructor, compute

similar to function invocation (take into similar to function invocation (take into account cost of pass by value parameters)account cost of pass by value parameters)

Jump statementsJump statements: continue/ break/ : continue/ break/ goto/ return / return<expr> :goto/ return / return<expr> : Step cost 1 for continue/break/goto/returnStep cost 1 for continue/break/goto/return return <expr>: Cost of <expr> return <expr>: Cost of <expr>

Measuring ComplexityMeasuring Complexity

First approach:First approach: Extend programs to incorporate step Extend programs to incorporate step

count statementscount statements Add global variable Add global variable countcount Whenever a statement is executed, add Whenever a statement is executed, add

an additional statement that increments an additional statement that increments countcount

Measuring ComplexityMeasuring Complexityfloat sum(float *a, const int n) {float sum(float *a, const int n) {

float s = 0;float s = 0;for (int i =0; i < n; i++) s += a[i];for (int i =0; i < n; i++) s += a[i];return s; }return s; }

float sum(float *a, const int n) {float sum(float *a, const int n) {float s = 0; float s = 0; count++;count++; // assignment// assignmentfor (int i =0; i < n; i++) for (int i =0; i < n; i++) { count++; s += a[i]; count++;}{ count++; s += a[i]; count++;} // 1 for // 1 for forfor , 1 , 1 for +=for +=count ++; // last time checking count ++; // last time checking forforreturn s; return s;

count ++; // return statementcount ++; // return statement}}


Strip out everything except count Strip out everything except count statements:statements:

float sum(float *a, const int n) {float sum(float *a, const int n) {

count++;count++; // assignment// assignment

for (int i =0; i < n; i++) for (int i =0; i < n; i++)

{ count = count + 2;}{ count = count + 2;} // 1 for for, 1 for +=// 1 for for, 1 for +=

// last time checking for and return statement// last time checking for and return statement

count = count + 2; count = count + 2;

}}

Measuring Complexity:Measuring Complexity:

Sum all count statements:Sum all count statements:1 at beginning1 at beginning

+ (N iterations of loop) * (2 within a loop)+ (N iterations of loop) * (2 within a loop)

+ 2 at end+ 2 at end

=> 2N + 3=> 2N + 3


float rsum(float *a, int n)float rsum(float *a, int n){{

if (n <= 0) return 0;if (n <= 0) return 0;else return (rsum(a, n-1) + a[n-1]);else return (rsum(a, n-1) + a[n-1]);

}}float rsum(float *a, int n)float rsum(float *a, int n){{

count ++; // if conditionalcount ++; // if conditionalif (n <= 0) { count++; return 0;} // returnif (n <= 0) { count++; return 0;} // returnelse { count++; return (rsum(a,n-1) + a[n-1]); else { count++; return (rsum(a,n-1) + a[n-1]); // return statement// return statement

}}

How to handle recursion?

Measuring Complexity: Measuring Complexity: RecursionRecursion

float rsum(float *a, cont int n)float rsum(float *a, cont int n){{

count ++; // if conditionalcount ++; // if conditionalif (n <= 0) { count++;} // returnif (n <= 0) { count++;} // returnelse { count++; return (rsum(a,n-1) + a[n-1]); else { count++; return (rsum(a,n-1) + a[n-1]);

}}

If (n <= 0) count = 2If (n <= 0) count = 2Otherwise count = 2 + count(n-1)Otherwise count = 2 + count(n-1)

Measuring Complexity: Measuring Complexity: RecursionRecursion

2 + count(n-1)2 + count(n-1) Recurrence relationRecurrence relation Solve by repeatedly substituting count(n-Solve by repeatedly substituting count(n-

1) with its recursive definition1) with its recursive definition

=> 2+ count(n-1) => 2+ count(n-1)

= 2 + 2 + count(n-2)= 2 + 2 + count(n-2)

= 2 + 2 + 2 + count(n – 3)= 2 + 2 + 2 + count(n – 3)

……

= 2 * n + count(0)= 2 * n + count(0) => 2n + 2=> 2n + 2


Iterative Sum: Iterative Sum: 2n + 32n + 3 Recursive Sum:Recursive Sum: 2n + 22n + 2

Is recursive sum faster than iterative?Is recursive sum faster than iterative?

Not necessarily – Each step of recursive sum may actually be more expensive than a step of iterative sum

Not a problem - We are really interested in measurements in relation to input size, which we see are very similar.


2n + 22n + 2 => => Run time is linear Run time is linear in Nin N

If input grows by a factor of 10, If input grows by a factor of 10, execution time should grow by a factor execution time should grow by a factor of 10of 10

If input grows by a factor of 10,000, If input grows by a factor of 10,000, cost should grow by a factor of 10,000cost should grow by a factor of 10,000

Xn + CXn + C => Xn often dominates => Xn often dominates constant term so we often ignore Cconstant term so we often ignore C


Matrix addition example:Matrix addition example:void add(matrix a, matrix b, matrix c, int m, int void add(matrix a, matrix b, matrix c, int m, int

n)n){{

for (int i = 0; i < m; i++)for (int i = 0; i < m; i++){{

for (int j = 0; j < n; j++)for (int j = 0; j < n; j++){{

c[i][j] = a[i][j] + b[i][j]c[i][j] = a[i][j] + b[i][j]}}

}}}}

Time ComplexityTime Complexityvoid add(matrix a, matrix b, matrix c, int m, int n)void add(matrix a, matrix b, matrix c, int m, int n){{

for (int i = 0; i < m; i++) {for (int i = 0; i < m; i++) {count++; // for loopcount++; // for loopfor (int j = 0; j < n; j++) {for (int j = 0; j < n; j++) {

count++; // for loopcount++; // for loopc[i][j] = a[i][j] + b[i][j]c[i][j] = a[i][j] + b[i][j]count++; // assignmentcount++; // assignment

}}count ++; // last time through for loop on jcount ++; // last time through for loop on j

}}count++; // last time through for loop on icount++; // last time through for loop on i

}}

Time ComplexityTime Complexityvoid add(matrix a, matrix b, matrix c, int m, int n)void add(matrix a, matrix b, matrix c, int m, int n){{

for (int i = 0; i < m; i++) for (int i = 0; i < m; i++) {{

count = count + 2; // for loop start, last time on j [case count = count + 2; // for loop start, last time on j [case 1]1]

for (int j = 0; j < n; j++) for (int j = 0; j < n; j++) { { count = count + 2; } // for loop start, assignment count = count + 2; } // for loop start, assignment

[case 2][case 2]}}

}}count++; // last time through for loop on i [case 3]count++; // last time through for loop on i [case 3]

}}=> 2m [case 1]+ 2mn [case 2] + 1 [case 3]=> 2m [case 1]+ 2mn [case 2] + 1 [case 3]


Matrix addition: 2mn + 2m + 1Matrix addition: 2mn + 2m + 1

What does that tell us?What does that tell us?

Potential speedup: If m >> n, then we Potential speedup: If m >> n, then we might want to switch order of loops, can might want to switch order of loops, can change 2m to 2n in the middlechange 2m to 2n in the middle

In general – dominated by 2mn statementIn general – dominated by 2mn statement


First approach: Add count First approach: Add count statementsstatements

Second approach: Tabulation Second approach: Tabulation methodmethod Steps per execution of each statementSteps per execution of each statement Frequency of each statementFrequency of each statement

Tabular Method for Tabular Method for ComplexityComplexity

float sum(float *a, const int float sum(float *a, const int n) n)

1 {1 {

22 float s = 0;float s = 0;

33 for (int i =0; i < n; i++) for (int i =0; i < n; i++)

44 s += a[i];s += a[i];

55 return s; return s;

6}6}

Line S/E Freq Total1 0 1

0 2 1 1

13 1 n+1 n+14 1 n

n5 1 1

16 0 1

0

Total: 2n+3

Tabular Method for Tabular Method for complexitycomplexity

float rsum(float *a, const int n)float rsum(float *a, const int n)

1 {1 {

22 if (n <= 0) return 0;if (n <= 0) return 0;

33 else return (rsum(a, n-1) + else return (rsum(a, n-1) + a[n-1]);a[n-1]);

4}4}

Line S/E Freq Total =0 >0 =0 >0

1 0 1 1 0 0 2 1 1 1 1

12b 1 1 0 1

03 1 + tn-1 0 1 0 1+tn-1

4 0 1 1 0 0

Total: n = 0 => 2 n > 0 => 2+ tn-1

Tabular Method for Tabular Method for ComplexityComplexity

void add(matrix a, matrix b, matrix c, int m, void add(matrix a, matrix b, matrix c, int m, int n)int n)

{{11 for (int i = 0; i < m; i++)for (int i = 0; i < m; i++)

{{22 for (int j = 0; j < n; j++)for (int j = 0; j < n; j++)

{{33 c[i][j] = a[i][j] + b[i][j]c[i][j] = a[i][j] + b[i][j]

}}}}

}}

Line S/E Freq Total1 1 m+1 m+1 2 1 m(n+1) mn+m3 1 mn mn

Total: 2mn + 2m + 1


Best-Case: Minimum number of Best-Case: Minimum number of steps that can be executed for the steps that can be executed for the given parametersgiven parameters

Worst-Case: Maximum number of Worst-Case: Maximum number of steps that can be executed for the steps that can be executed for the given parametersgiven parameters

Average-Case: Average number of Average-Case: Average number of steps that can be executed for the steps that can be executed for the given parametersgiven parameters


T(A) = cT(A) = c11nn22 + c + c22nn T(B) = cT(B) = c33nn

Consider cConsider c11 = 5, c = 5, c22 = 3, c = 3, c33 = 100 = 100When is cost of algorithm A > cost of algorithm When is cost of algorithm A > cost of algorithm

B?B?(When do we not want to use A anymore?)(When do we not want to use A anymore?)

5n5n22 + 3n > 100n + 3n > 100n => 5n=> 5n22 > 97n > 97n=> 5n > 97=> 5n > 97=> n > 19.4=> n > 19.4

19.4 is the 19.4 is the breakpointbreakpoint

Big “Oh” NotationBig “Oh” Notation

Defines relationships between Defines relationships between functionsfunctions

F(n) is O(g(n)) ifF(n) is O(g(n)) if Exists c and nExists c and n00 > 0 such that f(n) <= > 0 such that f(n) <=

cg(n) for all n, n >= ncg(n) for all n, n >= n00

3n + 2 is O(n) because:3n + 2 is O(n) because:

3n+2 <= 4n for all n >= 2 (c = 4, n3n+2 <= 4n for all n >= 2 (c = 4, n00 = 2)= 2)

Big “Oh” notationBig “Oh” notation

10n10n22 + 4n + 2 is O(n + 4n + 2 is O(n22) because:) because:

10n10n22 + 4n + 2 < 11n + 4n + 2 < 11n22 for all n >= 5 for all n >= 5 (c = 11, n(c = 11, n00 = 5) = 5)

10n10n22 + 4n + 2 is not O(n) because: + 4n + 2 is not O(n) because:

there do not exist any c or nthere do not exist any c or n0 0 for for which which

10n10n22 + 4n + 2 is < cn for all n >= n + 4n + 2 is < cn for all n >= n00

Big “Oh” notationBig “Oh” notationC = 100? C = 100? 10n10n22+4n+2 < 100n+4n+2 < 100n10n10n22 < 96n – 2 ~ 10n < 96n – 2 ~ 10n22 < 95n = 10n < 95 => n < 9.5 < 95n = 10n < 95 => n < 9.5If n = 1 through 9, inequality holds, but breaks around If n = 1 through 9, inequality holds, but breaks around

n = 10n = 10

C = 1000?C = 1000?10n10n22+4n+2 < 1000n+4n+2 < 1000n10n10n22 < 996n – 2 ~ 10n < 996n – 2 ~ 10n22 < 995n = 10n < 995 => n < < 995n = 10n < 995 => n <

99.599.5If n = 1 through 99, inequality holds, but breaks If n = 1 through 99, inequality holds, but breaks

around n = 100around n = 100

Can’t find a nCan’t find a n00 for which it holds for all n >= n for which it holds for all n >= n00


f(n) = O(g(n)) statement provides an f(n) = O(g(n)) statement provides an upper bound on the value of f(n)upper bound on the value of f(n)

Not necessarily the best upper Not necessarily the best upper bound:bound:Want to use minimum upper bound Want to use minimum upper bound you can findyou can find

F(n) = 4n + 2 is O(n) and O(nF(n) = 4n + 2 is O(n) and O(n22) and ) and O(nO(n33).. => Use O(n)).. => Use O(n)

Big “Oh” notationBig “Oh” notation

O(1)O(1) = = constantconstant O(n)O(n) = = linearlinear O(nO(n22)) = = quadraticquadratic O(nO(n33)) = = cubiccubic O(log n)O(log n) == logarithmiclogarithmic O(2O(2nn)) == exponentialexponential Commonly see algorithms that are O(n log Commonly see algorithms that are O(n log

n) as welln) as well


If a function has multiple terms with If a function has multiple terms with n, the first term dominates.n, the first term dominates.

f(n) = 4nf(n) = 4n55 + 6n + 6n33 + 2n + 2n22 + n + 8 => + n + 8 => O(nO(n55))

Omega NotationOmega Notation

f(n) = f(n) = ΩΩ(g(n)) iff(g(n)) iff Exists c and nExists c and n00 > 0 such that f(n) >= cg(n) > 0 such that f(n) >= cg(n)

for all n, n >= nfor all n, n >= n00

3n + 2 = 3n + 2 = Ω(n) because:Ω(n) because:

3n + 2 >= 3n for n >= 1 (c = 3, n = 1)3n + 2 >= 3n for n >= 1 (c = 3, n = 1) 10n10n22 + 4n + 2 = + 4n + 2 = Ω(nΩ(n22) because:) because:

10n10n22 + 4n + 2 >= n + 4n + 2 >= n22 for n >= 1 (c = 1, n for n >= 1 (c = 1, n = 1)= 1)

Also Also Ω(n), Ω(1)Ω(n), Ω(1)

Omega NotationOmega Notation

Ω(g(n)) provides a lower bound on Ω(g(n)) provides a lower bound on computation timecomputation time

Should use tightest lower bound you Should use tightest lower bound you can findcan find

Theta NotationTheta Notation f(n) = f(n) = Θ(g(n)) iff Θ(g(n)) iff

Exist positive constants cExist positive constants c11, c, c22, and n, and n00 such that such that cc11g(n) <= f(n) <= cg(n) <= f(n) <= c22g(n) for all n, n >= ng(n) for all n, n >= n00

3n + 2 is 3n + 2 is Θ(n) because:Θ(n) because:3n + 2 <= 4n for all n >= 23n + 2 <= 4n for all n >= 2 O(n)O(n)3n + 2 >= 3n for all n >= 23n + 2 >= 3n for all n >= 2 Ω(n)Ω(n)

Θ(g(n)) provides a tight upper and lower Θ(g(n)) provides a tight upper and lower bound on f(n)bound on f(n)

GraphsGraphs

Points to notice:Points to notice: What happens near the What happens near the

beginningbeginning((n n << N N) is not important) is not important

In the third diagram, In the third diagram, cc11g(n)g(n) and and cc22g(n)g(n) have the same have the same “shape” (why?)“shape” (why?)

f(n)

cg(n)

f(n) is O(g(n))

N

f(n)cg(n)

f(n) is (g(n))

N

f(n)c2g(n)

c1g(n)f(n) is (g(n))

Nhttp://www.cis.upenn.edu/~matuszek/cit594-2004/Lectures/49-analysis-2.ppt

Interesting ProblemsInteresting Problems

Problems with different O and Problems with different O and ΩΩThink of Big O: I know it requires at most this much Think of Big O: I know it requires at most this much

work, so if my algorithm is more than that, it’s a bad work, so if my algorithm is more than that, it’s a bad algorithmalgorithm

(Best known algorithm can provide this)(Best known algorithm can provide this)Think of Big Think of Big ΩΩ: I know it requires at least this much : I know it requires at least this much

work, so I should strive to get my algorithm to this work, so I should strive to get my algorithm to this point.point. (Theoretical proof over all algorithms usually (Theoretical proof over all algorithms usually provides this).provides this).

Would like to have same Big O and Big Would like to have same Big O and Big ΩΩ=> Then => Then you have optimal algorithm.you have optimal algorithm. There are some There are some algorithms where the Big O and Big algorithms where the Big O and Big ΩΩ are are different – These are ripe for improvement. different – These are ripe for improvement.

For self readingFor self reading

Computational Computational RequirementsRequirements

Require a clocking function with Require a clocking function with significant resolutionsignificant resolution

Decide on size of inputs Decide on size of inputs (n) (n) for for which times are to be obtainedwhich times are to be obtained

Determine, for each of the above Determine, for each of the above values of values of nn, data sets that exhibit , data sets that exhibit worst case behaviorworst case behavior

Timing FunctionTiming Function clock() functionclock() function

Returns number of clock ticks since start of program Returns number of clock ticks since start of program executionexecution

#include <time.h>#include <time.h> Tends to be more accurate than other time of day Tends to be more accurate than other time of day

functions because program dependent, not OS functions because program dependent, not OS dependent.dependent.

clock_t start, end;clock_t start, end;

double elapsed;double elapsed;

start = clock();start = clock();

… … do something …do something …

end = clock();end = clock();

elapsed = ((double) (end - start)) / CLOCKS_PER_SEC;elapsed = ((double) (end - start)) / CLOCKS_PER_SEC;

Choice of N to UseChoice of N to Use Assume we are measuring sequential search functionAssume we are measuring sequential search function Want to predict how long it will take, in worst case, to Want to predict how long it will take, in worst case, to

search for search for xx in an array of size in an array of size nn.. Find a graph and its corresponding function from dataFind a graph and its corresponding function from data

Asymptotic analysis: Asymptotic analysis: O(n)O(n) Expect plot of times to be a straight lineExpect plot of times to be a straight line

Asymptotic behavior only guaranteed to hold for Asymptotic behavior only guaranteed to hold for sufficiently large values of nsufficiently large values of n

In general, O(n) function may not always have points In general, O(n) function may not always have points end up on a line due to lower order costs (such as a end up on a line due to lower order costs (such as a log n)log n)

Choice of N to UseChoice of N to Use

Book suggests:Book suggests: Likely that asymptotic behavior will Likely that asymptotic behavior will

have started by size n = 100have started by size n = 100 Use a fairly separated set of points from Use a fairly separated set of points from

100+100+ Use a fairly refined set of points from 0 Use a fairly refined set of points from 0

to 100 to get good idea of low end to 100 to get good idea of low end behaviorbehavior

RepetitionRepetitionCan make up for poor timing resolution Can make up for poor timing resolution

through repeated execution timing blocks:through repeated execution timing blocks:

int ITERATIONS = 1000000;int ITERATIONS = 1000000;clock_t start, end;clock_t start, end;double sum = 0;double sum = 0;for (int i= 0; i < ITERATIONS; i++)for (int i= 0; i < ITERATIONS; i++){{

start = clock();start = clock(); … … work … work … end = clock();end = clock();sum += ((double) (end - start)) / CLOCKS_PER_SEC;sum += ((double) (end - start)) / CLOCKS_PER_SEC;

}}double averageTime = sum / ITERATIONS.double averageTime = sum / ITERATIONS.

Worst Case SelectionWorst Case Selection

Examine algorithm to determine Examine algorithm to determine inputs which provide worst case inputs which provide worst case performanceperformance

Sequential search: Sequential search: When item searched for is not present When item searched for is not present

in the array in the array Guarantees have to search entire arrayGuarantees have to search entire array

Worst Case SelectionWorst Case Selectionvoid insertionSort(int* a, int size)void insertionSort(int* a, int size){{

for (int k = 1; k < size; k++)for (int k = 1; k < size; k++){{

int temp = a[k];int temp = a[k];int position = k;int position = k;

while (position > 0 && a[position-1] > temp)while (position > 0 && a[position-1] > temp){{

a[position] = a[position-1];a[position] = a[position-1];position--;position--;

}}a[position] = temp;a[position] = temp;

}}}}


Insertion Sort Worst Case?Insertion Sort Worst Case?

Most comparisons occur when while Most comparisons occur when while loop runs all the way to position == 0loop runs all the way to position == 0

With what type of data would this With what type of data would this occur?occur?

Reverse SortedReverse Sorted

Worst Case SelectionWorst Case Selectionvoid selectionSort(int* a, int size)void selectionSort(int* a, int size){{

for (int k = 0; k < size-1; k++)for (int k = 0; k < size-1; k++){{

int index = mininumIndex(a, k, size-1);int index = mininumIndex(a, k, size-1);swap(a[k],a[index]);swap(a[k],a[index]);

}}}}

int minimumIndex(int* a, int first, int last)int minimumIndex(int* a, int first, int last){{

minIndex = first;minIndex = first;for (int j = first + 1; j <= last; j++)for (int j = first + 1; j <= last; j++){ if (a[j] < a[minIndex]) minIndex = j; }{ if (a[j] < a[minIndex]) minIndex = j; }return minIndex;return minIndex;

}}


Selection Sort Worst Case?Selection Sort Worst Case?

For comparisons, there is no case worse than For comparisons, there is no case worse than any other. There is no arrangement of data any other. There is no arrangement of data which changes how many comparisons the which changes how many comparisons the minIndex function performs, so the same minIndex function performs, so the same number of spots in the array are always number of spots in the array are always examined for a given input size.examined for a given input size.

However, reverse sorted data does force more However, reverse sorted data does force more assignments (everything is a new min).assignments (everything is a new min).


Adding Polynomials:Adding Polynomials:

3x3x33 + 2x + 1 + 2x + 1

++

2x2x33 + 3x + 3x22 + 5 + 5

==================================

5x5x33 + 3x + 3x22 + 2x + 6 + 2x + 6

PolynomialsPolynomials

Adding Polynomials:Adding Polynomials: Iterate through both listsIterate through both lists If exponent1 == exponent2,If exponent1 == exponent2,

CoefficientSum = Coefficient1 + Coefficient2CoefficientSum = Coefficient1 + Coefficient2 If CoefficentSum != 0, add term to new If CoefficentSum != 0, add term to new

polynomial representing answerpolynomial representing answer Else, Else,

Find higher exponentFind higher exponent Add term to new polynomial representing Add term to new polynomial representing

answeranswer

PolynomialsPolynomials

3 3 2 1 1 0

2 3 3 2 5 0

5 3 3 2 2 1 6 0


Polynomial addition worst case?Polynomial addition worst case?

All exponents are different and All exponents are different and interleaved. Requires O(m+n) exponent interleaved. Requires O(m+n) exponent comparisons, where m, n are list sizes.comparisons, where m, n are list sizes.

Best you can do? All same exponents – Best you can do? All same exponents – Requires O(m) comparisons - which is Requires O(m) comparisons - which is size of either list size of either list (have to be same size to have all same (have to be same size to have all same exponents).exponents).

Worst Case/Average Case Worst Case/Average Case GenerationGeneration

Not always possible to generate worst Not always possible to generate worst case data or average case data by handcase data or average case data by hand Average case – often times way too many Average case – often times way too many

problem instance to test them all and do problem instance to test them all and do averageaverage

Best approach: Generate a large number Best approach: Generate a large number of random instancesof random instances Worst case – Worst overall time out of Worst case – Worst overall time out of

instancesinstances Average case – Average over instance timesAverage case – Average over instance times

Analysis of Search Analysis of Search FunctionsFunctions

seqsearch.cpp seqsearch.cpp programprogram

Sequential Search vs Binary Search

0.00E+00

5.00E-05

1.00E-04

1.50E-04

2.00E-04

2.50E-04

0 10000 20000 30000 40000 50000 60000 70000 80000 90000 100000

Array Size

Av

era

ge

Se

arc

h T

ime

Sequential AverageSequential WorstBinary AverageBinary Worst


Binary Search: Average vs Worst Case

0.00E+00

2.00E-07

4.00E-07

6.00E-07

8.00E-07

1.00E-06

1.20E-06

1.40E-06

1.60E-06

0 10000 20000 30000 40000 50000 60000 70000 80000 90000 100000

Array Size

Av

era

ge

Se

arc

h T

ime

Binary Average

Binary Worst


Sequential vs Binary Search: Small Size Problems

0.00E+00

1.00E-06

2.00E-06

3.00E-06

4.00E-06

5.00E-06

6.00E-06

7.00E-06

0 100

200

300

400

500

600

700

800

900

1000

1100

1200

1300

1400

1500

1600

1700

1800

1900

2000

Array Size

Ave

rag

e S

earc

h T

ime

Sequential Average

Sequential Worst

Binary Average

Binary Worst

Analysis of SortingAnalysis of Sorting

Selection vs. Insertion Sort

0.00E+00

1.00E-03

2.00E-03

3.00E-03

4.00E-03

5.00E-03

6.00E-03

0 200 400 600 800 1000

Array Size

Ave

rag

e S

ort

Tim

e

SelectionSelection WorstInsertionInsertion Worst

Recursion. Recursion: Definition Function that solves a problem by relying on itself to compute the...

Documents

Transcript of Recursion. Recursion: Definition Function that solves a problem by relying on itself to compute the...