Recursion Colin Capham Recursion – an introduction to the concept.
Recursion and Dynamic Programming - University of...
Transcript of Recursion and Dynamic Programming - University of...
![Page 1: Recursion and Dynamic Programming - University of …csg.sph.umich.edu/abecasis/class/2006/615.05.pdf · Key Applications of Recursion zDynamic Programming • Related to Markov processes](https://reader031.fdocuments.net/reader031/viewer/2022022511/5ae066aa7f8b9afd1a8de46c/html5/thumbnails/1.jpg)
Recursion and Dynamic Programming
Biostatistics 615/815Lecture 5
![Page 2: Recursion and Dynamic Programming - University of …csg.sph.umich.edu/abecasis/class/2006/615.05.pdf · Key Applications of Recursion zDynamic Programming • Related to Markov processes](https://reader031.fdocuments.net/reader031/viewer/2022022511/5ae066aa7f8b9afd1a8de46c/html5/thumbnails/2.jpg)
Last Lecture
Principles for analysis of algorithms• Empirical Analysis• Theoretical Analysis
Common relationships between inputs and running time
Described two simple search algorithms
![Page 3: Recursion and Dynamic Programming - University of …csg.sph.umich.edu/abecasis/class/2006/615.05.pdf · Key Applications of Recursion zDynamic Programming • Related to Markov processes](https://reader031.fdocuments.net/reader031/viewer/2022022511/5ae066aa7f8b9afd1a8de46c/html5/thumbnails/3.jpg)
Recursive refers to …
A function that is part of its own definition
e.g.
A program that calls itself
⎪⎩
⎪⎨
⎧
=
>−⋅
=
0 N if1
0 N if)1()(
NFactorialNNFactorial
![Page 4: Recursion and Dynamic Programming - University of …csg.sph.umich.edu/abecasis/class/2006/615.05.pdf · Key Applications of Recursion zDynamic Programming • Related to Markov processes](https://reader031.fdocuments.net/reader031/viewer/2022022511/5ae066aa7f8b9afd1a8de46c/html5/thumbnails/4.jpg)
Key Applications of Recursion
Dynamic Programming• Related to Markov processes in Statistics
Divide-and-Conquer Algorithms
Tree Processing
![Page 5: Recursion and Dynamic Programming - University of …csg.sph.umich.edu/abecasis/class/2006/615.05.pdf · Key Applications of Recursion zDynamic Programming • Related to Markov processes](https://reader031.fdocuments.net/reader031/viewer/2022022511/5ae066aa7f8b9afd1a8de46c/html5/thumbnails/5.jpg)
Recursive Function in C
int factorial (int N){if (N == 0)return 1;
elsereturn N * factorial(N - 1);
}
![Page 6: Recursion and Dynamic Programming - University of …csg.sph.umich.edu/abecasis/class/2006/615.05.pdf · Key Applications of Recursion zDynamic Programming • Related to Markov processes](https://reader031.fdocuments.net/reader031/viewer/2022022511/5ae066aa7f8b9afd1a8de46c/html5/thumbnails/6.jpg)
Key Features of Recursions
Simple solution for a few cases
Recursive definition for other values• Computation of large N depends on smaller N
Can be naturally expressed in a function that calls itself• Loops are sometimes an alternative
![Page 7: Recursion and Dynamic Programming - University of …csg.sph.umich.edu/abecasis/class/2006/615.05.pdf · Key Applications of Recursion zDynamic Programming • Related to Markov processes](https://reader031.fdocuments.net/reader031/viewer/2022022511/5ae066aa7f8b9afd1a8de46c/html5/thumbnails/7.jpg)
A Typical Recursion:Euclid’s Algorithm
Algorithm for finding greatest common divisor of two integers a and b• If a divides b
• GCD(a,b) is a
• Otherwise, find the largest integer t such that• at + r = b• GCD(a,b) = GCD(r,a)
![Page 8: Recursion and Dynamic Programming - University of …csg.sph.umich.edu/abecasis/class/2006/615.05.pdf · Key Applications of Recursion zDynamic Programming • Related to Markov processes](https://reader031.fdocuments.net/reader031/viewer/2022022511/5ae066aa7f8b9afd1a8de46c/html5/thumbnails/8.jpg)
Euclid’s Algorithm in C
int gcd (int a, int b){if (a == 0)return b;
return gcd(b % a, a);}
![Page 9: Recursion and Dynamic Programming - University of …csg.sph.umich.edu/abecasis/class/2006/615.05.pdf · Key Applications of Recursion zDynamic Programming • Related to Markov processes](https://reader031.fdocuments.net/reader031/viewer/2022022511/5ae066aa7f8b9afd1a8de46c/html5/thumbnails/9.jpg)
Evaluating GCD(4458, 2099)
gcd(2099, 4458)gcd(350, 2099)
gcd(349, 350)gcd(1, 349)
gcd(0, 1)
![Page 10: Recursion and Dynamic Programming - University of …csg.sph.umich.edu/abecasis/class/2006/615.05.pdf · Key Applications of Recursion zDynamic Programming • Related to Markov processes](https://reader031.fdocuments.net/reader031/viewer/2022022511/5ae066aa7f8b9afd1a8de46c/html5/thumbnails/10.jpg)
Divide-And-Conquer Algorithms
Common class of recursive functions
Common feature• Process input• Divide input in smaller portions• Recursive call(s) process at least one portion
Recursion may sometimes occur before input is processed
![Page 11: Recursion and Dynamic Programming - University of …csg.sph.umich.edu/abecasis/class/2006/615.05.pdf · Key Applications of Recursion zDynamic Programming • Related to Markov processes](https://reader031.fdocuments.net/reader031/viewer/2022022511/5ae066aa7f8b9afd1a8de46c/html5/thumbnails/11.jpg)
Recursive Binary Searchint search(int a[], int value, int start, int stop)
{// Search failedif (start > stop)
return -1;
// Find midpointint mid = (start + stop) / 2;
// Compare midpoint to valueif (value == a[mid]) return mid;
// Reduce input in half!!!if (value < a[mid])
return search(a, start, mid – 1);else
return search(a, mid + 1, stop);}
![Page 12: Recursion and Dynamic Programming - University of …csg.sph.umich.edu/abecasis/class/2006/615.05.pdf · Key Applications of Recursion zDynamic Programming • Related to Markov processes](https://reader031.fdocuments.net/reader031/viewer/2022022511/5ae066aa7f8b9afd1a8de46c/html5/thumbnails/12.jpg)
Recursive Maximumint Maximum(int a[], int start, int stop)
{int left, right;
// Maximum of one elementif (start == stop)
return a[start];
left = Maximum(a, start, (start + stop) / 2);right = Maximum(a, (start + stop) / 2 + 1, stop);
// Reduce input in half!!!if (left > right)
return left;else
return right;}
![Page 13: Recursion and Dynamic Programming - University of …csg.sph.umich.edu/abecasis/class/2006/615.05.pdf · Key Applications of Recursion zDynamic Programming • Related to Markov processes](https://reader031.fdocuments.net/reader031/viewer/2022022511/5ae066aa7f8b9afd1a8de46c/html5/thumbnails/13.jpg)
An inefficient recursion
Consider the Fibonacci numbers…
⎪⎪⎪
⎩
⎪⎪⎪
⎨
⎧
−+−
=
=
=
)2()1(
1 N if1
0 N if0
)(
NFibonacciNFibonacci
NFibonacci
![Page 14: Recursion and Dynamic Programming - University of …csg.sph.umich.edu/abecasis/class/2006/615.05.pdf · Key Applications of Recursion zDynamic Programming • Related to Markov processes](https://reader031.fdocuments.net/reader031/viewer/2022022511/5ae066aa7f8b9afd1a8de46c/html5/thumbnails/14.jpg)
Fibonacci Numbersint Fibonacci(int i)
{// Simple cases firstif (i == 0)
return 0;
if (i == 1)return 1;
return Fibonacci(i – 1) + Fibonacci(i – 2);}
![Page 15: Recursion and Dynamic Programming - University of …csg.sph.umich.edu/abecasis/class/2006/615.05.pdf · Key Applications of Recursion zDynamic Programming • Related to Markov processes](https://reader031.fdocuments.net/reader031/viewer/2022022511/5ae066aa7f8b9afd1a8de46c/html5/thumbnails/15.jpg)
Terribly Slow!
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32
Time
0
10
20
30
40
50
60
Fibonacci Number
Time (seconds)Calculating Fibonacci Numbers Recursively
![Page 16: Recursion and Dynamic Programming - University of …csg.sph.umich.edu/abecasis/class/2006/615.05.pdf · Key Applications of Recursion zDynamic Programming • Related to Markov processes](https://reader031.fdocuments.net/reader031/viewer/2022022511/5ae066aa7f8b9afd1a8de46c/html5/thumbnails/16.jpg)
What is going on? …
![Page 17: Recursion and Dynamic Programming - University of …csg.sph.umich.edu/abecasis/class/2006/615.05.pdf · Key Applications of Recursion zDynamic Programming • Related to Markov processes](https://reader031.fdocuments.net/reader031/viewer/2022022511/5ae066aa7f8b9afd1a8de46c/html5/thumbnails/17.jpg)
Faster Alternatives
Certain quantities are recalculated• Far too many times!
Need to avoid recalculation…• Ideally, calculate each unique quantity once.
![Page 18: Recursion and Dynamic Programming - University of …csg.sph.umich.edu/abecasis/class/2006/615.05.pdf · Key Applications of Recursion zDynamic Programming • Related to Markov processes](https://reader031.fdocuments.net/reader031/viewer/2022022511/5ae066aa7f8b9afd1a8de46c/html5/thumbnails/18.jpg)
Dynamic Programming
A technique for avoiding recomputation
Can make exponential running times …
… become linear!
![Page 19: Recursion and Dynamic Programming - University of …csg.sph.umich.edu/abecasis/class/2006/615.05.pdf · Key Applications of Recursion zDynamic Programming • Related to Markov processes](https://reader031.fdocuments.net/reader031/viewer/2022022511/5ae066aa7f8b9afd1a8de46c/html5/thumbnails/19.jpg)
Bottom-Up Dynamic Programming
Evaluate function starting with smallest possible argument value• Stepping through possible values, gradually increase
argument value
Store all computed values in an array
As larger arguments evaluated, precomputed values for smaller arguments can be retrieved
![Page 20: Recursion and Dynamic Programming - University of …csg.sph.umich.edu/abecasis/class/2006/615.05.pdf · Key Applications of Recursion zDynamic Programming • Related to Markov processes](https://reader031.fdocuments.net/reader031/viewer/2022022511/5ae066aa7f8b9afd1a8de46c/html5/thumbnails/20.jpg)
Fibonacci Numbers in Cint Fibonacci(int i)
{int fib[LARGE_NUMBER], j;
fib[0] = 0;fib[1] = 1;
for (j = 2; j <= i; j++)fib[j] = fib[j – 1] + fib[j – 2];
return fib[i];}
![Page 21: Recursion and Dynamic Programming - University of …csg.sph.umich.edu/abecasis/class/2006/615.05.pdf · Key Applications of Recursion zDynamic Programming • Related to Markov processes](https://reader031.fdocuments.net/reader031/viewer/2022022511/5ae066aa7f8b9afd1a8de46c/html5/thumbnails/21.jpg)
Fibonacci With Dynamic Memoryint Fibonacci(int i)
{int * fib, j, result;
if (i < 2) return i;
fib = malloc(sizeof(int) * (i + 1));
fib[0] = 0; fib[1] = 1;for (j = 2; j <= i; j++)
fib[j] = fib[j – 1] + fib[j – 2];
result = fib[i];free(fib);
return result;}
![Page 22: Recursion and Dynamic Programming - University of …csg.sph.umich.edu/abecasis/class/2006/615.05.pdf · Key Applications of Recursion zDynamic Programming • Related to Markov processes](https://reader031.fdocuments.net/reader031/viewer/2022022511/5ae066aa7f8b9afd1a8de46c/html5/thumbnails/22.jpg)
Top-Down Dynamic Programming
Save each computed value as final action of recursive function
Check if pre-computed value exists as the first action
![Page 23: Recursion and Dynamic Programming - University of …csg.sph.umich.edu/abecasis/class/2006/615.05.pdf · Key Applications of Recursion zDynamic Programming • Related to Markov processes](https://reader031.fdocuments.net/reader031/viewer/2022022511/5ae066aa7f8b9afd1a8de46c/html5/thumbnails/23.jpg)
Fibonacci Numbers// Note: saveF should be a global array initialized all zeros
int Fibonacci(int i){// Simple cases firstif (saveF[i] > 0)
return saveF[i];
if (i <= 1)return i;
// RecursionsaveF[i] = Fibonacci(i – 1) + Fibonacci(i – 2);return saveF[i];}
![Page 24: Recursion and Dynamic Programming - University of …csg.sph.umich.edu/abecasis/class/2006/615.05.pdf · Key Applications of Recursion zDynamic Programming • Related to Markov processes](https://reader031.fdocuments.net/reader031/viewer/2022022511/5ae066aa7f8b9afd1a8de46c/html5/thumbnails/24.jpg)
Much less recursion now…
![Page 25: Recursion and Dynamic Programming - University of …csg.sph.umich.edu/abecasis/class/2006/615.05.pdf · Key Applications of Recursion zDynamic Programming • Related to Markov processes](https://reader031.fdocuments.net/reader031/viewer/2022022511/5ae066aa7f8b9afd1a8de46c/html5/thumbnails/25.jpg)
Dynamic ProgrammingTop-down vs. Bottom-up
In bottom-up programming, programmer has to do the thinking by selecting values to calculate and order of calculation
In top-down programming, recursive structure of original code is preserved, but unnecessary recalculation is avoided.
![Page 26: Recursion and Dynamic Programming - University of …csg.sph.umich.edu/abecasis/class/2006/615.05.pdf · Key Applications of Recursion zDynamic Programming • Related to Markov processes](https://reader031.fdocuments.net/reader031/viewer/2022022511/5ae066aa7f8b9afd1a8de46c/html5/thumbnails/26.jpg)
Examples of Useful Settings for Dynamic Programming
Calculating Binomial Coefficients
Evaluating Poisson-Binomial Distribution
![Page 27: Recursion and Dynamic Programming - University of …csg.sph.umich.edu/abecasis/class/2006/615.05.pdf · Key Applications of Recursion zDynamic Programming • Related to Markov processes](https://reader031.fdocuments.net/reader031/viewer/2022022511/5ae066aa7f8b9afd1a8de46c/html5/thumbnails/27.jpg)
Binomial Coefficients
The number of subsets with k elements from a set of size N
⎟⎟⎠
⎞⎜⎜⎝
⎛−−
+⎟⎟⎠
⎞⎜⎜⎝
⎛ −=⎟⎟
⎠
⎞⎜⎜⎝
⎛111
kN
kN
kN
10
=⎟⎟⎠
⎞⎜⎜⎝
⎛=⎟⎟
⎠
⎞⎜⎜⎝
⎛NNN
![Page 28: Recursion and Dynamic Programming - University of …csg.sph.umich.edu/abecasis/class/2006/615.05.pdf · Key Applications of Recursion zDynamic Programming • Related to Markov processes](https://reader031.fdocuments.net/reader031/viewer/2022022511/5ae066aa7f8b9afd1a8de46c/html5/thumbnails/28.jpg)
Bottom-Up Implementation in Cint Choose(int N, int k)
{int i, j, M[MAX_N][MAX_N];
for (i = 1; i <= N; i++){M[i][0] = M[i][i] = 1;
for (j = 1; j < i; j++)M[i][j] = M[i - 1][j - 1] + M[i - 1][j];
}
return M[N][k];}
![Page 29: Recursion and Dynamic Programming - University of …csg.sph.umich.edu/abecasis/class/2006/615.05.pdf · Key Applications of Recursion zDynamic Programming • Related to Markov processes](https://reader031.fdocuments.net/reader031/viewer/2022022511/5ae066aa7f8b9afd1a8de46c/html5/thumbnails/29.jpg)
Top-Down Implementation (I)#define MAX_N 30
int choices[MAX_N][MAX_N];
void InitChoose(){int i, j;
for (i = 0; i < MAX_N; i++)for (j = 0; j < MAX_N; j++)
choices[i][j] = 0;}
![Page 30: Recursion and Dynamic Programming - University of …csg.sph.umich.edu/abecasis/class/2006/615.05.pdf · Key Applications of Recursion zDynamic Programming • Related to Markov processes](https://reader031.fdocuments.net/reader031/viewer/2022022511/5ae066aa7f8b9afd1a8de46c/html5/thumbnails/30.jpg)
Top-Down Implementation (II)int Choose(int N, int k)
{// Check if this is an easy caseif (N == k || k == 0)
return 1;
// Or a previously examined caseif (choices[N][k] > 0)
return choices[N][k];
// If neither of the above helps, use recursionchoices[N][k] = Choose(N – 1, k - 1) + Choose(N – 1, k);
return choices[N][k];}
![Page 31: Recursion and Dynamic Programming - University of …csg.sph.umich.edu/abecasis/class/2006/615.05.pdf · Key Applications of Recursion zDynamic Programming • Related to Markov processes](https://reader031.fdocuments.net/reader031/viewer/2022022511/5ae066aa7f8b9afd1a8de46c/html5/thumbnails/31.jpg)
Poisson-Binomial Distribution
X1, X2, …, Xn are Bernoulli random variables
Probability of success is pk for Xk
∑kXk has Poisson-Binomial Distribution
![Page 32: Recursion and Dynamic Programming - University of …csg.sph.umich.edu/abecasis/class/2006/615.05.pdf · Key Applications of Recursion zDynamic Programming • Related to Markov processes](https://reader031.fdocuments.net/reader031/viewer/2022022511/5ae066aa7f8b9afd1a8de46c/html5/thumbnails/32.jpg)
Recursive Formulation
)()1()1()(
)1()(
)0()1()0(
)1(1)0(
11
1
1
11
11
iPpiPpiP
jPpjP
PpP
pPpP
jjjjj
jjj
jjj
−−
−
−
−+−=
−=
−=
=−=
![Page 33: Recursion and Dynamic Programming - University of …csg.sph.umich.edu/abecasis/class/2006/615.05.pdf · Key Applications of Recursion zDynamic Programming • Related to Markov processes](https://reader031.fdocuments.net/reader031/viewer/2022022511/5ae066aa7f8b9afd1a8de46c/html5/thumbnails/33.jpg)
Summary
Recursive functions• Arise very often in statistics
Dynamic programming• Bottom-up Dynamic Programming• Top-down Dynamic Programming
Dynamic program is an essential tool for statistical programming
![Page 34: Recursion and Dynamic Programming - University of …csg.sph.umich.edu/abecasis/class/2006/615.05.pdf · Key Applications of Recursion zDynamic Programming • Related to Markov processes](https://reader031.fdocuments.net/reader031/viewer/2022022511/5ae066aa7f8b9afd1a8de46c/html5/thumbnails/34.jpg)
Good Programming Practices
Today’s examples used global variables• Variables declared outside a function• Accessible throughout the program
In general, these should be used sparingly
Two alternatives are:• Using static variables that are setup on the first call• Using C++ to group a function and its data
![Page 35: Recursion and Dynamic Programming - University of …csg.sph.umich.edu/abecasis/class/2006/615.05.pdf · Key Applications of Recursion zDynamic Programming • Related to Markov processes](https://reader031.fdocuments.net/reader031/viewer/2022022511/5ae066aa7f8b9afd1a8de46c/html5/thumbnails/35.jpg)
Function with Built-In Initializationint Choose(int N, int k)
{static int valid = 0, choices[MAX_N][MAX_N], i, j;
// Check if we need to initialize dataif (valid == 0)
{for (i = 0; i < MAX_N; i++)
for (j = 0; j < MAX_N; j++)choices[i][j] = 0;
valid = 1; }
// Check if this is an easy caseif (N == k || k == 0) return 1;
// Or a previously examined caseif (choices[N][k] > 0) return choices[N][k];
// If neither of the above helps, use recursionchoices[N][k] = Choose(N – 1, k - 1) + Choose(N – 1, k);
return choices[N][k];}
![Page 36: Recursion and Dynamic Programming - University of …csg.sph.umich.edu/abecasis/class/2006/615.05.pdf · Key Applications of Recursion zDynamic Programming • Related to Markov processes](https://reader031.fdocuments.net/reader031/viewer/2022022511/5ae066aa7f8b9afd1a8de46c/html5/thumbnails/36.jpg)
C++ Declarations (for .h file)class Choose
{public:
// This function is called to initialize data// Whenever a variable of type Choose is created Choose();
// This function is called to calculate N_choose_kint Evaluate(int N, int k);
private:int choices[MAX_N][MAX_N];
};
![Page 37: Recursion and Dynamic Programming - University of …csg.sph.umich.edu/abecasis/class/2006/615.05.pdf · Key Applications of Recursion zDynamic Programming • Related to Markov processes](https://reader031.fdocuments.net/reader031/viewer/2022022511/5ae066aa7f8b9afd1a8de46c/html5/thumbnails/37.jpg)
C++ Code (for .cpp file)Choose::Choose()
{for (int i = 0; i < MAX_N; i++)
for (int j = 0; j < MAX_N; j++)choices[i][j] = 0;
}
int Choose::Evaluate(){// Check if this is an easy caseif (N == k || k == 0) return 1;
// Or a previously examined caseif (choices[N][k] > 0) return choices[N][k];
// If neither of the above helps, use recursionchoices[N][k] = Choose(N – 1, k - 1) + Choose(N – 1, k);
return choices[N][k];}
![Page 38: Recursion and Dynamic Programming - University of …csg.sph.umich.edu/abecasis/class/2006/615.05.pdf · Key Applications of Recursion zDynamic Programming • Related to Markov processes](https://reader031.fdocuments.net/reader031/viewer/2022022511/5ae066aa7f8b9afd1a8de46c/html5/thumbnails/38.jpg)
Using C++ Code#include “Choose.h”
int main(){Choose choices;
// Evaluate 20_choose_10choices.Evaluate(20, 10);
// Evaluate 30_choose_10choices.Evaluate(30, 10);
return 0;}
![Page 39: Recursion and Dynamic Programming - University of …csg.sph.umich.edu/abecasis/class/2006/615.05.pdf · Key Applications of Recursion zDynamic Programming • Related to Markov processes](https://reader031.fdocuments.net/reader031/viewer/2022022511/5ae066aa7f8b9afd1a8de46c/html5/thumbnails/39.jpg)
Reading
Sedgewick, Chapters 5.1 – 5.3