Introduction to Parallel Processing with Python...What is Parallel Computing? 3 Parallel Computing:...
Transcript of Introduction to Parallel Processing with Python...What is Parallel Computing? 3 Parallel Computing:...
![Page 1: Introduction to Parallel Processing with Python...What is Parallel Computing? 3 Parallel Computing: Breaking a problem into multiple pieces and processing each piece in parallel through](https://reader035.fdocuments.net/reader035/viewer/2022062223/60ac8b47ff04930e45039bcd/html5/thumbnails/1.jpg)
Introduction to Parallel Processing with Python
1
![Page 2: Introduction to Parallel Processing with Python...What is Parallel Computing? 3 Parallel Computing: Breaking a problem into multiple pieces and processing each piece in parallel through](https://reader035.fdocuments.net/reader035/viewer/2022062223/60ac8b47ff04930e45039bcd/html5/thumbnails/2.jpg)
2 Source - https://computing.llnl.gov/tutorials/parallel_comp/
What is Parallel Computing?
Serial Computing
![Page 3: Introduction to Parallel Processing with Python...What is Parallel Computing? 3 Parallel Computing: Breaking a problem into multiple pieces and processing each piece in parallel through](https://reader035.fdocuments.net/reader035/viewer/2022062223/60ac8b47ff04930e45039bcd/html5/thumbnails/3.jpg)
What is Parallel Computing?
3
Parallel Computing: Breaking a problem into multiple pieces and processing each piece in parallel through multiple processors
![Page 4: Introduction to Parallel Processing with Python...What is Parallel Computing? 3 Parallel Computing: Breaking a problem into multiple pieces and processing each piece in parallel through](https://reader035.fdocuments.net/reader035/viewer/2022062223/60ac8b47ff04930e45039bcd/html5/thumbnails/4.jpg)
Parallelized Hardware
4
Nearly all processors now have parallelized processing
architectures
Eight-core CPUs on now selling for mainstream consumers
Intel® Core™ i7-5960X: $1000 (2014)
AMD Ryzen 2700X: $300 (2018)
![Page 5: Introduction to Parallel Processing with Python...What is Parallel Computing? 3 Parallel Computing: Breaking a problem into multiple pieces and processing each piece in parallel through](https://reader035.fdocuments.net/reader035/viewer/2022062223/60ac8b47ff04930e45039bcd/html5/thumbnails/5.jpg)
HPCs – Built for Parallelization
5
• HPCs employ often 2-4 server-grade CPUs per node• 8 – 16 processor cores per CPU• Shared memory on each node for all processors
• Distributed memory architecture• Nodes are connected via a 56-100 Gbps network• Memory is shared between nodes through some API
• MPI is most commonly used
![Page 6: Introduction to Parallel Processing with Python...What is Parallel Computing? 3 Parallel Computing: Breaking a problem into multiple pieces and processing each piece in parallel through](https://reader035.fdocuments.net/reader035/viewer/2022062223/60ac8b47ff04930e45039bcd/html5/thumbnails/6.jpg)
Global Interpreter Lock
6
![Page 7: Introduction to Parallel Processing with Python...What is Parallel Computing? 3 Parallel Computing: Breaking a problem into multiple pieces and processing each piece in parallel through](https://reader035.fdocuments.net/reader035/viewer/2022062223/60ac8b47ff04930e45039bcd/html5/thumbnails/7.jpg)
Global Interpreter Lock
7
• The Python interpreter is not fully thread-safe.
• In order to support multi-threaded Python programs, there’s a global lock, called the global interpreter lock or GIL, that must be held by the current thread before it can safely access Python objects.
• Without the lock, even the simplest operations could cause problems in a multi-threaded program
• For example, when two threads simultaneously increment the reference count of the same object, the reference count could end up being incremented only once instead of twice.
• Therefore, only one thread is run at a time.
![Page 8: Introduction to Parallel Processing with Python...What is Parallel Computing? 3 Parallel Computing: Breaking a problem into multiple pieces and processing each piece in parallel through](https://reader035.fdocuments.net/reader035/viewer/2022062223/60ac8b47ff04930e45039bcd/html5/thumbnails/8.jpg)
So how can one effectively parallelize their code?
8
Enter: multiprocessing
Time to switch over to Jupyter Notebook
![Page 9: Introduction to Parallel Processing with Python...What is Parallel Computing? 3 Parallel Computing: Breaking a problem into multiple pieces and processing each piece in parallel through](https://reader035.fdocuments.net/reader035/viewer/2022062223/60ac8b47ff04930e45039bcd/html5/thumbnails/9.jpg)
Installing a Conda Environment for Keras and TensorFlow with Jupyter Support
9
$ module load python/3.6.1-2-anaconda
$ conda create --name py3.6-multiprocess -–clone root$ source activate py3.6-multiprocess$ conda install –c conda-forge multiprocess
$ ipython kernel install --user --name py3.6-multiprocess --display-name=“Custom"