INTRODUCTION TO ALGORITHMS: COMPUTATIONAL...
Transcript of INTRODUCTION TO ALGORITHMS: COMPUTATIONAL...
![Page 1: INTRODUCTION TO ALGORITHMS: COMPUTATIONAL COMPLEXITYvaxxxa.github.io/talks/introduction.to.algorithms... · 2016-08-04 · 5. To select best algorithm, compare their T(n) functions.](https://reader033.fdocuments.net/reader033/viewer/2022060311/5f0af11f7e708231d42e18c3/html5/thumbnails/1.jpg)
INTRODUCTION TOALGORITHMS:
COMPUTATIONALCOMPLEXITY
BY VASYL NAKVASIUK, 2013
![Page 2: INTRODUCTION TO ALGORITHMS: COMPUTATIONAL COMPLEXITYvaxxxa.github.io/talks/introduction.to.algorithms... · 2016-08-04 · 5. To select best algorithm, compare their T(n) functions.](https://reader033.fdocuments.net/reader033/viewer/2022060311/5f0af11f7e708231d42e18c3/html5/thumbnails/2.jpg)
WHAT IS ANALGORITHM?
![Page 3: INTRODUCTION TO ALGORITHMS: COMPUTATIONAL COMPLEXITYvaxxxa.github.io/talks/introduction.to.algorithms... · 2016-08-04 · 5. To select best algorithm, compare their T(n) functions.](https://reader033.fdocuments.net/reader033/viewer/2022060311/5f0af11f7e708231d42e18c3/html5/thumbnails/3.jpg)
WHAT IS AN ALGORITHM?
Important issues: correctness, elegance and efficiency.
An algorithm is a procedure that takes any ofthe possible input instances and transforms it
to the desired output.
![Page 4: INTRODUCTION TO ALGORITHMS: COMPUTATIONAL COMPLEXITYvaxxxa.github.io/talks/introduction.to.algorithms... · 2016-08-04 · 5. To select best algorithm, compare their T(n) functions.](https://reader033.fdocuments.net/reader033/viewer/2022060311/5f0af11f7e708231d42e18c3/html5/thumbnails/4.jpg)
EFFICIENCYIs this really necessary?
![Page 5: INTRODUCTION TO ALGORITHMS: COMPUTATIONAL COMPLEXITYvaxxxa.github.io/talks/introduction.to.algorithms... · 2016-08-04 · 5. To select best algorithm, compare their T(n) functions.](https://reader033.fdocuments.net/reader033/viewer/2022060311/5f0af11f7e708231d42e18c3/html5/thumbnails/5.jpg)
CRITERIA OF EFFICIENCY:Time complexitySpace complexity
Time complexity ≠ Space complexity ≠ Complexity ofalgorithm
![Page 6: INTRODUCTION TO ALGORITHMS: COMPUTATIONAL COMPLEXITYvaxxxa.github.io/talks/introduction.to.algorithms... · 2016-08-04 · 5. To select best algorithm, compare their T(n) functions.](https://reader033.fdocuments.net/reader033/viewer/2022060311/5f0af11f7e708231d42e18c3/html5/thumbnails/6.jpg)
HOW CAN WE MEASURECOMPLEXITY?
![Page 7: INTRODUCTION TO ALGORITHMS: COMPUTATIONAL COMPLEXITYvaxxxa.github.io/talks/introduction.to.algorithms... · 2016-08-04 · 5. To select best algorithm, compare their T(n) functions.](https://reader033.fdocuments.net/reader033/viewer/2022060311/5f0af11f7e708231d42e18c3/html5/thumbnails/7.jpg)
HOW CAN WE MEASURE COMPLEXITY?
EMPIRICAL ANALYSIS (BENCHMARKS)THEORETICAL ANALYSIS (ASYMPTOTIC ANALYSIS)
![Page 8: INTRODUCTION TO ALGORITHMS: COMPUTATIONAL COMPLEXITYvaxxxa.github.io/talks/introduction.to.algorithms... · 2016-08-04 · 5. To select best algorithm, compare their T(n) functions.](https://reader033.fdocuments.net/reader033/viewer/2022060311/5f0af11f7e708231d42e18c3/html5/thumbnails/8.jpg)
BENCHMARKSEMPIRICAL ANALYSIS
![Page 9: INTRODUCTION TO ALGORITHMS: COMPUTATIONAL COMPLEXITYvaxxxa.github.io/talks/introduction.to.algorithms... · 2016-08-04 · 5. To select best algorithm, compare their T(n) functions.](https://reader033.fdocuments.net/reader033/viewer/2022060311/5f0af11f7e708231d42e18c3/html5/thumbnails/9.jpg)
BENCHMARKSVERSION #1
WHAT MEANS “FAST”?
![Page 10: INTRODUCTION TO ALGORITHMS: COMPUTATIONAL COMPLEXITYvaxxxa.github.io/talks/introduction.to.algorithms... · 2016-08-04 · 5. To select best algorithm, compare their T(n) functions.](https://reader033.fdocuments.net/reader033/viewer/2022060311/5f0af11f7e708231d42e18c3/html5/thumbnails/10.jpg)
BENCHMARKSVERSION #2
import time
start = time.time() # Return the time in seconds since the epoch.my_algo(some_input)end = time.time()
print(end - start)
0.048032498359680176
![Page 11: INTRODUCTION TO ALGORITHMS: COMPUTATIONAL COMPLEXITYvaxxxa.github.io/talks/introduction.to.algorithms... · 2016-08-04 · 5. To select best algorithm, compare their T(n) functions.](https://reader033.fdocuments.net/reader033/viewer/2022060311/5f0af11f7e708231d42e18c3/html5/thumbnails/11.jpg)
BENCHMARKSVERSION #3
import timeit
timeit.timeit('my_algo(some_input)', number=1000)
1000 loops, best of 3: 50.3 ms per loop
![Page 12: INTRODUCTION TO ALGORITHMS: COMPUTATIONAL COMPLEXITYvaxxxa.github.io/talks/introduction.to.algorithms... · 2016-08-04 · 5. To select best algorithm, compare their T(n) functions.](https://reader033.fdocuments.net/reader033/viewer/2022060311/5f0af11f7e708231d42e18c3/html5/thumbnails/12.jpg)
BENCHMARKSVERSION #4
import timeit
inputs = [1000, 10000, 500000, 1000000]
for input in inputs: timeit.timeit('my_algo(input)', number=1000)
list of 1000 items:1000 loops, best of 3: 50.3 ms per loop
list of 10000 items:1000 loops, best of 3: 104.7 ms per loop
list of 500000 items:1000 loops, best of 3: 459.1 ms per loop
list of 1000000 items:1000 loops, best of 3: 3.12 s per loop
![Page 13: INTRODUCTION TO ALGORITHMS: COMPUTATIONAL COMPLEXITYvaxxxa.github.io/talks/introduction.to.algorithms... · 2016-08-04 · 5. To select best algorithm, compare their T(n) functions.](https://reader033.fdocuments.net/reader033/viewer/2022060311/5f0af11f7e708231d42e18c3/html5/thumbnails/13.jpg)
BENCHMARKSVERSION #5
# Intel Core i7-3970X @ 3.50GHz, RAM 8 Gb, Ubuntu 12.10 x64, Python 3.3.0
import timeit
inputs = [1000, 10000, 500000, 1000000]
for input in inputs: timeit.timeit('my_algo(input)', number=1000)
list of 1000 items:1000 loops, best of 3: 50.3 ms per loop
list of 10000 items:1000 loops, best of 3: 104.7 ms per loop
list of 500000 items:1000 loops, best of 3: 459.1 ms per loop
list of 1000000 items:1000 loops, best of 3: 3.12 s per loop
![Page 14: INTRODUCTION TO ALGORITHMS: COMPUTATIONAL COMPLEXITYvaxxxa.github.io/talks/introduction.to.algorithms... · 2016-08-04 · 5. To select best algorithm, compare their T(n) functions.](https://reader033.fdocuments.net/reader033/viewer/2022060311/5f0af11f7e708231d42e18c3/html5/thumbnails/14.jpg)
EXPERIMENTAL STUDIES HAVE SEVERALLIMITATIONS:
It is necessary to implement and test the algorithm in orderto determine its running time.Experiments can be done only on a limited set of inputs, andmay not be indicative of the running time on other inputsnot included in the experiment.In order to compare two algorithms, the same hardware andsoftware environments should be used.
![Page 15: INTRODUCTION TO ALGORITHMS: COMPUTATIONAL COMPLEXITYvaxxxa.github.io/talks/introduction.to.algorithms... · 2016-08-04 · 5. To select best algorithm, compare their T(n) functions.](https://reader033.fdocuments.net/reader033/viewer/2022060311/5f0af11f7e708231d42e18c3/html5/thumbnails/15.jpg)
ASYMPTOTIC ANALYSISTHEORETICAL ANALYSIS
![Page 16: INTRODUCTION TO ALGORITHMS: COMPUTATIONAL COMPLEXITYvaxxxa.github.io/talks/introduction.to.algorithms... · 2016-08-04 · 5. To select best algorithm, compare their T(n) functions.](https://reader033.fdocuments.net/reader033/viewer/2022060311/5f0af11f7e708231d42e18c3/html5/thumbnails/16.jpg)
ASYMPTOTIC ANALYSISEFFICIENCY AS A FUNCTION OF INPUT SIZE
T(n) – running time as a function of n, where n – size of input.
n → ∞
Random-Access Machine (RAM)
![Page 17: INTRODUCTION TO ALGORITHMS: COMPUTATIONAL COMPLEXITYvaxxxa.github.io/talks/introduction.to.algorithms... · 2016-08-04 · 5. To select best algorithm, compare their T(n) functions.](https://reader033.fdocuments.net/reader033/viewer/2022060311/5f0af11f7e708231d42e18c3/html5/thumbnails/17.jpg)
BEST, WORST, AND AVERAGE-CASECOMPLEXITY
LINEAR SEARCH
T(n) = n ?
T(n) = 1/2 ⋅ n ?
T(n) = 1 ?
def linear_search(my_item, items): for position, item in enumerate(items): if my_item == item: return position
![Page 18: INTRODUCTION TO ALGORITHMS: COMPUTATIONAL COMPLEXITYvaxxxa.github.io/talks/introduction.to.algorithms... · 2016-08-04 · 5. To select best algorithm, compare their T(n) functions.](https://reader033.fdocuments.net/reader033/viewer/2022060311/5f0af11f7e708231d42e18c3/html5/thumbnails/18.jpg)
BEST, WORST, AND AVERAGE-CASECOMPLEXITY
![Page 19: INTRODUCTION TO ALGORITHMS: COMPUTATIONAL COMPLEXITYvaxxxa.github.io/talks/introduction.to.algorithms... · 2016-08-04 · 5. To select best algorithm, compare their T(n) functions.](https://reader033.fdocuments.net/reader033/viewer/2022060311/5f0af11f7e708231d42e18c3/html5/thumbnails/19.jpg)
BEST, WORST, AND AVERAGE-CASECOMPLEXITY
LINEAR SEARCH
Worst case: T(n) = n
Average case: T(n) = 1/2 ⋅ n
Best case: T(n) = 1
T(n) = O(n)
def linear_search(my_item, items): for position, item in enumerate(items): if my_item == item: return position
![Page 20: INTRODUCTION TO ALGORITHMS: COMPUTATIONAL COMPLEXITYvaxxxa.github.io/talks/introduction.to.algorithms... · 2016-08-04 · 5. To select best algorithm, compare their T(n) functions.](https://reader033.fdocuments.net/reader033/viewer/2022060311/5f0af11f7e708231d42e18c3/html5/thumbnails/20.jpg)
HOW CAN WE COMPARETWO FUNCTIONS?
WE CAN USEASYMPTOTIC NOTATION
![Page 21: INTRODUCTION TO ALGORITHMS: COMPUTATIONAL COMPLEXITYvaxxxa.github.io/talks/introduction.to.algorithms... · 2016-08-04 · 5. To select best algorithm, compare their T(n) functions.](https://reader033.fdocuments.net/reader033/viewer/2022060311/5f0af11f7e708231d42e18c3/html5/thumbnails/21.jpg)
ASYMPTOTIC NOTATION
![Page 22: INTRODUCTION TO ALGORITHMS: COMPUTATIONAL COMPLEXITYvaxxxa.github.io/talks/introduction.to.algorithms... · 2016-08-04 · 5. To select best algorithm, compare their T(n) functions.](https://reader033.fdocuments.net/reader033/viewer/2022060311/5f0af11f7e708231d42e18c3/html5/thumbnails/22.jpg)
THE BIG OH NOTATIONASYMPTOTIC UPPER BOUND
O(g(n)) = {f(n): there exist positive constants c and n0 such that
0 ≤ f(n) ≤ c⋅g(n) for all n ≥ n0}
T(n) ∈ O(g(n))
or
T(n) = O(g(n))
![Page 23: INTRODUCTION TO ALGORITHMS: COMPUTATIONAL COMPLEXITYvaxxxa.github.io/talks/introduction.to.algorithms... · 2016-08-04 · 5. To select best algorithm, compare their T(n) functions.](https://reader033.fdocuments.net/reader033/viewer/2022060311/5f0af11f7e708231d42e18c3/html5/thumbnails/23.jpg)
Ω-NOTATIONASYMPTOTIC LOWER BOUND
Ω(g(n)) = {f(n): there exist positive constants c and n0 such that
0 ≤ c⋅g(n) ≤ f(n) for all n ≥ n0}
T(n) ∈ Ω(g(n))
or
T(n) = Ω(g(n))
![Page 24: INTRODUCTION TO ALGORITHMS: COMPUTATIONAL COMPLEXITYvaxxxa.github.io/talks/introduction.to.algorithms... · 2016-08-04 · 5. To select best algorithm, compare their T(n) functions.](https://reader033.fdocuments.net/reader033/viewer/2022060311/5f0af11f7e708231d42e18c3/html5/thumbnails/24.jpg)
Θ-NOTATIONASYMPTOTIC TIGHT BOUND
Θ(g(n)) = {f(n): there exist positive constants c1, c2 and n0 such
that 0 ≤ c1⋅g(n) ≤ f(n) ≤ c2⋅g(n) for all n ≥ n0}
T(n) ∈ Θ(g(n))
or
T(n) = Θ(g(n))
![Page 25: INTRODUCTION TO ALGORITHMS: COMPUTATIONAL COMPLEXITYvaxxxa.github.io/talks/introduction.to.algorithms... · 2016-08-04 · 5. To select best algorithm, compare their T(n) functions.](https://reader033.fdocuments.net/reader033/viewer/2022060311/5f0af11f7e708231d42e18c3/html5/thumbnails/25.jpg)
GRAPHIC EXAMPLES OF THE Θ, O AND ΩNOTATIONS
![Page 26: INTRODUCTION TO ALGORITHMS: COMPUTATIONAL COMPLEXITYvaxxxa.github.io/talks/introduction.to.algorithms... · 2016-08-04 · 5. To select best algorithm, compare their T(n) functions.](https://reader033.fdocuments.net/reader033/viewer/2022060311/5f0af11f7e708231d42e18c3/html5/thumbnails/26.jpg)
EXAMPLES
3⋅n2 - 100⋅n + 6 = O(n2),because we can choose c = 3 and
3⋅n2 > 3⋅n2 - 100⋅n + 6
100⋅n2 - 70⋅n - 1 = O(n2),because we can choose c = 100 and
100⋅n2 > 100⋅n2 - 70⋅n - 1
3⋅n2 - 100⋅n + 6 ≈ 100⋅n2 - 70⋅n - 1
![Page 27: INTRODUCTION TO ALGORITHMS: COMPUTATIONAL COMPLEXITYvaxxxa.github.io/talks/introduction.to.algorithms... · 2016-08-04 · 5. To select best algorithm, compare their T(n) functions.](https://reader033.fdocuments.net/reader033/viewer/2022060311/5f0af11f7e708231d42e18c3/html5/thumbnails/27.jpg)
LINEAR SEARCHLINEAR SEARCH (VILLARRIBA VERSION):
T(n) = O(n)
LINEAR SEARCH (VILLABAJO VERSION)
T(n) = O(3⋅n + 2) = O(n)
Speed of "Villarriba version" ≈ Speed of "Villabajo version"
def linear_search(my_item, items): for position, item in enumerate(items): print('position – {0}, item – {0}'.format(position, item)) print('Compare two items.') if my_item == item: print('Yeah!!!') print('The end!') return position
![Page 28: INTRODUCTION TO ALGORITHMS: COMPUTATIONAL COMPLEXITYvaxxxa.github.io/talks/introduction.to.algorithms... · 2016-08-04 · 5. To select best algorithm, compare their T(n) functions.](https://reader033.fdocuments.net/reader033/viewer/2022060311/5f0af11f7e708231d42e18c3/html5/thumbnails/28.jpg)
TYPES OF ORDER
However, all you really need to understand is that:
n! ≫ 2n ≫ n3 ≫ n2 ≫ n⋅log(n) ≫ n ≫ log(n) ≫ 1
![Page 29: INTRODUCTION TO ALGORITHMS: COMPUTATIONAL COMPLEXITYvaxxxa.github.io/talks/introduction.to.algorithms... · 2016-08-04 · 5. To select best algorithm, compare their T(n) functions.](https://reader033.fdocuments.net/reader033/viewer/2022060311/5f0af11f7e708231d42e18c3/html5/thumbnails/29.jpg)
THE BIG OH COMPLEXITY FOR DIFFERENTFUNCTIONS
![Page 30: INTRODUCTION TO ALGORITHMS: COMPUTATIONAL COMPLEXITYvaxxxa.github.io/talks/introduction.to.algorithms... · 2016-08-04 · 5. To select best algorithm, compare their T(n) functions.](https://reader033.fdocuments.net/reader033/viewer/2022060311/5f0af11f7e708231d42e18c3/html5/thumbnails/30.jpg)
GROWTH RATES OF COMMON FUNCTIONSMEASURED IN NANOSECONDS
Each operation takes one nanosecond (10-9 seconds).
CPU ≈ 1 GHz
![Page 31: INTRODUCTION TO ALGORITHMS: COMPUTATIONAL COMPLEXITYvaxxxa.github.io/talks/introduction.to.algorithms... · 2016-08-04 · 5. To select best algorithm, compare their T(n) functions.](https://reader033.fdocuments.net/reader033/viewer/2022060311/5f0af11f7e708231d42e18c3/html5/thumbnails/31.jpg)
BINARY SEARCH
T(n) = O(log(n))
def binary_search(seq, t): min = 0; max = len(seq) - 1 while 1: if max < min: return -1 m = (min + max) / 2 if seq[m] < t: min = m + 1 elif seq[m] > t: max = m - 1 else: return m
![Page 32: INTRODUCTION TO ALGORITHMS: COMPUTATIONAL COMPLEXITYvaxxxa.github.io/talks/introduction.to.algorithms... · 2016-08-04 · 5. To select best algorithm, compare their T(n) functions.](https://reader033.fdocuments.net/reader033/viewer/2022060311/5f0af11f7e708231d42e18c3/html5/thumbnails/32.jpg)
PRACTICAL USAGEADD DB “INDEX”
Search with index vs Search without index
Binary search vs Linear search
O(log(n)) vs O(n)
![Page 33: INTRODUCTION TO ALGORITHMS: COMPUTATIONAL COMPLEXITYvaxxxa.github.io/talks/introduction.to.algorithms... · 2016-08-04 · 5. To select best algorithm, compare their T(n) functions.](https://reader033.fdocuments.net/reader033/viewer/2022060311/5f0af11f7e708231d42e18c3/html5/thumbnails/33.jpg)
HOW CAN YOU QUICKLY FIND OUTCOMPLEXITY?
O(?)
![Page 34: INTRODUCTION TO ALGORITHMS: COMPUTATIONAL COMPLEXITYvaxxxa.github.io/talks/introduction.to.algorithms... · 2016-08-04 · 5. To select best algorithm, compare their T(n) functions.](https://reader033.fdocuments.net/reader033/viewer/2022060311/5f0af11f7e708231d42e18c3/html5/thumbnails/34.jpg)
D. E. Knuth, "Big Omicron and Big Omega and BIg Theta", SIGACT News, 1976.
On the basis of the issues discussed here, Ipropose that members of SIGACT, and editors
of computer science and mathematicsjournals, adopt the O, Ω and Θ notations as
defined above, unless a better alternative canbe found reasonably soon.
![Page 35: INTRODUCTION TO ALGORITHMS: COMPUTATIONAL COMPLEXITYvaxxxa.github.io/talks/introduction.to.algorithms... · 2016-08-04 · 5. To select best algorithm, compare their T(n) functions.](https://reader033.fdocuments.net/reader033/viewer/2022060311/5f0af11f7e708231d42e18c3/html5/thumbnails/35.jpg)
BENCHMARKSOR
ASYMPTOTIC ANALYSIS?USE BOTH APPROACHES!
![Page 36: INTRODUCTION TO ALGORITHMS: COMPUTATIONAL COMPLEXITYvaxxxa.github.io/talks/introduction.to.algorithms... · 2016-08-04 · 5. To select best algorithm, compare their T(n) functions.](https://reader033.fdocuments.net/reader033/viewer/2022060311/5f0af11f7e708231d42e18c3/html5/thumbnails/36.jpg)
SUMMARY1. We want to predict running time of an algorithm.2. Summarize all possible inputs with a single “size” parameter
n.3. Many problems with “empirical” approach (measure lots of
test cases with various n and then extrapolate).4. Prefer “analytical” approach.5. To select best algorithm, compare their T(n) functions.6. To simplify this comparision “round” the function using
asymptotic (“big-O”) notation7. Amazing fact: Even though asymptotic complexity analysis
makes many simplifying assumptions, it is remarkably useful
in practice: if A is O(n3) and B is O(n2) then B really will befaster than A, no matter how they’re implemented.
![Page 37: INTRODUCTION TO ALGORITHMS: COMPUTATIONAL COMPLEXITYvaxxxa.github.io/talks/introduction.to.algorithms... · 2016-08-04 · 5. To select best algorithm, compare their T(n) functions.](https://reader033.fdocuments.net/reader033/viewer/2022060311/5f0af11f7e708231d42e18c3/html5/thumbnails/37.jpg)
LINKSBOOKS:
“Introduction To Algorithms, Third Edition”, 2009, byThomas H. Cormen, Charles E. Leiserson, Ronald L. Rivestand Clifford Stein“The Algorithm Design Manual, Second Edition”, 2008, bySteven S. Skiena
OTHER:
“Algorithms: Design and Analysis” by Tim Roughgarden
Big-O Algorithm Complexity Cheat Sheet https://www.coursera.org/course/algo
http://bigocheatsheet.com/
![Page 38: INTRODUCTION TO ALGORITHMS: COMPUTATIONAL COMPLEXITYvaxxxa.github.io/talks/introduction.to.algorithms... · 2016-08-04 · 5. To select best algorithm, compare their T(n) functions.](https://reader033.fdocuments.net/reader033/viewer/2022060311/5f0af11f7e708231d42e18c3/html5/thumbnails/38.jpg)
THE END
THANK YOU FOR ATTENTION!Vasyl NakvasiukEmail: [email protected]: @vaxXxaGithub: vaxXxa
THIS PRESENTATION:Source:
Live:
https://github.com/vaxXxa/talks
http://vaxXxa.github.io/talks