High-Performance Python
-
Upload
work-bench -
Category
Data & Analytics
-
view
7.374 -
download
0
Transcript of High-Performance Python
![Page 1: High-Performance Python](https://reader031.fdocuments.net/reader031/viewer/2022021421/58f046761a28ab4c568b45b5/html5/thumbnails/1.jpg)
High-Performance Python
![Page 2: High-Performance Python](https://reader031.fdocuments.net/reader031/viewer/2022021421/58f046761a28ab4c568b45b5/html5/thumbnails/2.jpg)
Python is fast!
• Python is fast to write, but natively 10x - 100x slower than C.
• Python has great C interop, so you can use C for the slow parts.
• This makes Python competitive with C.
![Page 3: High-Performance Python](https://reader031.fdocuments.net/reader031/viewer/2022021421/58f046761a28ab4c568b45b5/html5/thumbnails/3.jpg)
Before you try this at home…
• “Premature optimization is the root of all evil.”
• Use external standards for how fast your code needs to be.
• Remember: performance is a tradeoff against readability, maintainability, and developer time.
![Page 4: High-Performance Python](https://reader031.fdocuments.net/reader031/viewer/2022021421/58f046761a28ab4c568b45b5/html5/thumbnails/4.jpg)
Part 1: General Optimization
![Page 5: High-Performance Python](https://reader031.fdocuments.net/reader031/viewer/2022021421/58f046761a28ab4c568b45b5/html5/thumbnails/5.jpg)
Profile Your Code
• 95%+ of your code is irrelevant to performance.
• A profiler will tells you which 5% is important.
![Page 6: High-Performance Python](https://reader031.fdocuments.net/reader031/viewer/2022021421/58f046761a28ab4c568b45b5/html5/thumbnails/6.jpg)
Profile Your CodeIn Python, use cProfile:
source: https://ymichael.com/2014/03/08/profiling-python-with-cprofile.html
![Page 7: High-Performance Python](https://reader031.fdocuments.net/reader031/viewer/2022021421/58f046761a28ab4c568b45b5/html5/thumbnails/7.jpg)
Basics• Make sure your Big-O performance is optimal.
• Move operations outside of loops.
• Use cacheing for repeated calculations.
• Apply algebraic simplifications.
![Page 8: High-Performance Python](https://reader031.fdocuments.net/reader031/viewer/2022021421/58f046761a28ab4c568b45b5/html5/thumbnails/8.jpg)
Accidentally QuadraticThe *most* common issue:
def find_intersection(list_one, list_two): intersection = [] for a in list_one: if a in list_two: intersection.append(a) return intersection
![Page 9: High-Performance Python](https://reader031.fdocuments.net/reader031/viewer/2022021421/58f046761a28ab4c568b45b5/html5/thumbnails/9.jpg)
Accidentally QuadraticThe *most* common issue:
def find_intersection(list_one, list_two): intersection = [] for a in list_one: if a in list_two: intersection.append(a) return intersection
def find_intersection(list_one, list_two): intersection = [] list_two = set(list_two) for a in list_one: if a in list_two: intersection.append(a) return intersection
![Page 10: High-Performance Python](https://reader031.fdocuments.net/reader031/viewer/2022021421/58f046761a28ab4c568b45b5/html5/thumbnails/10.jpg)
Business Logic
Leverage business logic. You’ll often have NP-Complete optimizations to make.
The underlying business reasoning should guide your approximations.
![Page 11: High-Performance Python](https://reader031.fdocuments.net/reader031/viewer/2022021421/58f046761a28ab4c568b45b5/html5/thumbnails/11.jpg)
Part II: Python Optimization
![Page 12: High-Performance Python](https://reader031.fdocuments.net/reader031/viewer/2022021421/58f046761a28ab4c568b45b5/html5/thumbnails/12.jpg)
Libraries• Use numpy, scipy, pandas, scikit-learn, etc.
• Incredible built-in functionality. If you need something esoteric, try combining built-ins or adapting a more general built-in approach.
• Extremely fast, thoroughly optimized, and best of all, already written.
![Page 13: High-Performance Python](https://reader031.fdocuments.net/reader031/viewer/2022021421/58f046761a28ab4c568b45b5/html5/thumbnails/13.jpg)
Pure Python Tips
• Function calls are expensive. Avoid them and avoid recursion.
• Check the runtime of built-in data types.
• Make variables local. Global lookups are expensive.
• Use map/filter/reduce instead of for loops, they’re written in C.
![Page 14: High-Performance Python](https://reader031.fdocuments.net/reader031/viewer/2022021421/58f046761a28ab4c568b45b5/html5/thumbnails/14.jpg)
• Vectorize! numpy arrays are much faster than lists.
Mixed Tips
![Page 15: High-Performance Python](https://reader031.fdocuments.net/reader031/viewer/2022021421/58f046761a28ab4c568b45b5/html5/thumbnails/15.jpg)
• Vectorize! numpy arrays are much faster than lists.
Mixed Tips
def complex_sum(in_list): in_list = [(a + 2) for a in in_list] # more transformations
return sum(in_list)
def complex_sum(in_list): in_list = np.array(in_list) in_list += 2 # more transformations
return in_list.sum()
![Page 16: High-Performance Python](https://reader031.fdocuments.net/reader031/viewer/2022021421/58f046761a28ab4c568b45b5/html5/thumbnails/16.jpg)
Mixed Tips• Vectorize! numpy arrays are much faster than lists.
• Array allocation can be a bottleneck. Try moving it outside of loops.
![Page 17: High-Performance Python](https://reader031.fdocuments.net/reader031/viewer/2022021421/58f046761a28ab4c568b45b5/html5/thumbnails/17.jpg)
Mixed Tips• Vectorize! numpy arrays are much faster than lists.
• Array allocation can be a bottleneck. Try moving it outside of loops.
n = 10 ** 3 output = 0
for i in xrange(10**9): result = np.zeros(n) ## calculations ## output += result.sum()
result = np.zeros(10**3) output = 0
for i in xrange(10**9): result[:] = 0 # zero out array ## calculations ## output += result.sum()
![Page 18: High-Performance Python](https://reader031.fdocuments.net/reader031/viewer/2022021421/58f046761a28ab4c568b45b5/html5/thumbnails/18.jpg)
• Cython: inline C code directly into Python.
Last Resort: C
![Page 19: High-Performance Python](https://reader031.fdocuments.net/reader031/viewer/2022021421/58f046761a28ab4c568b45b5/html5/thumbnails/19.jpg)
def fib(int n): cdef int a, b, temp a = 0 b = 1 while b < n: temp = b b = a + b a = temp
• Cython: inline C code directly into Python.
Last Resort: C
def fib(n): a = 0 b = 1 while b < n: temp = b b = a + b a = temp return b
![Page 20: High-Performance Python](https://reader031.fdocuments.net/reader031/viewer/2022021421/58f046761a28ab4c568b45b5/html5/thumbnails/20.jpg)
• Cython: inline C code directly into Python.
Last Resort: C
def fib(int n): cdef int a, b, temp a = 0 b = 1 while b < n: temp = b b = a + b a = temp return b
![Page 21: High-Performance Python](https://reader031.fdocuments.net/reader031/viewer/2022021421/58f046761a28ab4c568b45b5/html5/thumbnails/21.jpg)
Last Resort: C
• Cython: inline C code directly into Python.
• C extensions: write C and call it from Python.
![Page 22: High-Performance Python](https://reader031.fdocuments.net/reader031/viewer/2022021421/58f046761a28ab4c568b45b5/html5/thumbnails/22.jpg)
Last Resort: C
• Cython: inline C code directly into Python.
• C extensions: write C and call it from Python.
• Limit these techniques to hot loops.
![Page 23: High-Performance Python](https://reader031.fdocuments.net/reader031/viewer/2022021421/58f046761a28ab4c568b45b5/html5/thumbnails/23.jpg)
Things I haven’t mentioned
• multithreading: basically doesn’t work in Python
• pypy: A Python JIT compiler with a different ecosystem
![Page 24: High-Performance Python](https://reader031.fdocuments.net/reader031/viewer/2022021421/58f046761a28ab4c568b45b5/html5/thumbnails/24.jpg)
Warning
Optimization is addictive.
![Page 25: High-Performance Python](https://reader031.fdocuments.net/reader031/viewer/2022021421/58f046761a28ab4c568b45b5/html5/thumbnails/25.jpg)
Conclusions• Avoid premature optimizations!
Have objective benchmarks you’re trying to hit.
• Profile your code. You will be surprised by the results.
• The gold standard for performance is highly-tuned C (that’s already been written by someone else)
![Page 26: High-Performance Python](https://reader031.fdocuments.net/reader031/viewer/2022021421/58f046761a28ab4c568b45b5/html5/thumbnails/26.jpg)
Resources
• Programming Pearls (Jon Bentley)
• accidentallyquadratic.tumblr.com
• Performance Engineering of Software Systems, 6.172, MIT OpenCourseWare
• cProfile Docs
• Cython Docs
• Guido Van Rossum’s advice:python.org/doc/essays/list2str
General Python Specific
Contact me: [email protected]