University of Delaware Dept of Computer & Information Sciences · OpenMP: Environment Variables...
Transcript of University of Delaware Dept of Computer & Information Sciences · OpenMP: Environment Variables...
![Page 1: University of Delaware Dept of Computer & Information Sciences · OpenMP: Environment Variables OpenMP has a set of environment variables that control the runtime execution OMP_NUM_THREADS=num](https://reader035.fdocuments.net/reader035/viewer/2022063012/5fc7a21a32634908943f3e8c/html5/thumbnails/1.jpg)
John Cavazos and Tristan VanderbruggenDept of Computer & Information Sciences
University of Delaware
Parallelism II
1
![Page 2: University of Delaware Dept of Computer & Information Sciences · OpenMP: Environment Variables OpenMP has a set of environment variables that control the runtime execution OMP_NUM_THREADS=num](https://reader035.fdocuments.net/reader035/viewer/2022063012/5fc7a21a32634908943f3e8c/html5/thumbnails/2.jpg)
Lecture Overview
● Introduction ● OpenMP
○ Model○ Language extension: directives-based○ Step-by-step example
● MPI○ Model○ Runtime Library○ Step-by-step example
● Conclusion / Q&A2
![Page 3: University of Delaware Dept of Computer & Information Sciences · OpenMP: Environment Variables OpenMP has a set of environment variables that control the runtime execution OMP_NUM_THREADS=num](https://reader035.fdocuments.net/reader035/viewer/2022063012/5fc7a21a32634908943f3e8c/html5/thumbnails/3.jpg)
● Codes:○ http://www.cis.udel.edu/~cavazos/hpc-II.zip
or○ https://github.com/tristanvdb/hpc-lecture
● To connect on Mills and start using it:○ $> ssh [email protected]○ $> workgroup -g your_workgroup○ $> vpkg_devrequire gcc○ $> vpkg_devrequire openmpi
Accessing Mills and the codes
3
![Page 4: University of Delaware Dept of Computer & Information Sciences · OpenMP: Environment Variables OpenMP has a set of environment variables that control the runtime execution OMP_NUM_THREADS=num](https://reader035.fdocuments.net/reader035/viewer/2022063012/5fc7a21a32634908943f3e8c/html5/thumbnails/4.jpg)
![Page 5: University of Delaware Dept of Computer & Information Sciences · OpenMP: Environment Variables OpenMP has a set of environment variables that control the runtime execution OMP_NUM_THREADS=num](https://reader035.fdocuments.net/reader035/viewer/2022063012/5fc7a21a32634908943f3e8c/html5/thumbnails/5.jpg)
2 - OpenMP
● Model● Language● Step-by-step Example● Construct & Clause● Q&A
4
![Page 6: University of Delaware Dept of Computer & Information Sciences · OpenMP: Environment Variables OpenMP has a set of environment variables that control the runtime execution OMP_NUM_THREADS=num](https://reader035.fdocuments.net/reader035/viewer/2022063012/5fc7a21a32634908943f3e8c/html5/thumbnails/6.jpg)
2.1 - OpenMP: Model
5
● Shared Memory Model:○ multi-processor/core
Source: https://computing.llnl.gov/tutorials/openMP/
![Page 7: University of Delaware Dept of Computer & Information Sciences · OpenMP: Environment Variables OpenMP has a set of environment variables that control the runtime execution OMP_NUM_THREADS=num](https://reader035.fdocuments.net/reader035/viewer/2022063012/5fc7a21a32634908943f3e8c/html5/thumbnails/7.jpg)
2.1 - OpenMP: Model
6
● Thread-level Parallelism:○ parallelism through threads○ typically: the number of threads match the number
of cores● Fork - Join Model:
Source: https://computing.llnl.gov/tutorials/openMP/
![Page 8: University of Delaware Dept of Computer & Information Sciences · OpenMP: Environment Variables OpenMP has a set of environment variables that control the runtime execution OMP_NUM_THREADS=num](https://reader035.fdocuments.net/reader035/viewer/2022063012/5fc7a21a32634908943f3e8c/html5/thumbnails/8.jpg)
2.1 - OpenMP: Model
7Source: https://computing.llnl.gov/tutorials/openMP/
● Explicit Parallelism:○ offer full control over parallelization to the
programmer○ can be as simple as inserting compiler directives in
a serial program○ or, as complex as inserting subroutines to set
multiple levels of parallelism, locks and even nested locks
![Page 9: University of Delaware Dept of Computer & Information Sciences · OpenMP: Environment Variables OpenMP has a set of environment variables that control the runtime execution OMP_NUM_THREADS=num](https://reader035.fdocuments.net/reader035/viewer/2022063012/5fc7a21a32634908943f3e8c/html5/thumbnails/9.jpg)
2.2 - OpenMP: Language
8
● OpenMP is not exactly a language. It is an extension for C and Fortran.
● OpenMP is a directive-based language● It works by annotating sequential code
Source: https://computing.llnl.gov/tutorials/openMP/
![Page 10: University of Delaware Dept of Computer & Information Sciences · OpenMP: Environment Variables OpenMP has a set of environment variables that control the runtime execution OMP_NUM_THREADS=num](https://reader035.fdocuments.net/reader035/viewer/2022063012/5fc7a21a32634908943f3e8c/html5/thumbnails/10.jpg)
2.2 - OpenMP: Language
9
● in C, it uses pragmas
#pragma omp construct [clause, ...]
● in Fortran, it uses sentinels (!$omp, C$omp, or *$omp):
!$OMP construct [clause, ...]
Source: https://computing.llnl.gov/tutorials/openMP/
![Page 11: University of Delaware Dept of Computer & Information Sciences · OpenMP: Environment Variables OpenMP has a set of environment variables that control the runtime execution OMP_NUM_THREADS=num](https://reader035.fdocuments.net/reader035/viewer/2022063012/5fc7a21a32634908943f3e8c/html5/thumbnails/11.jpg)
2.2 - OpenMP: Language (cont'd)
10Source: https://computing.llnl.gov/tutorials/openMP/
● constructs are functionalities of the language
● clauses are parameters to those functionalities
● construct + clauses = directive
![Page 12: University of Delaware Dept of Computer & Information Sciences · OpenMP: Environment Variables OpenMP has a set of environment variables that control the runtime execution OMP_NUM_THREADS=num](https://reader035.fdocuments.net/reader035/viewer/2022063012/5fc7a21a32634908943f3e8c/html5/thumbnails/12.jpg)
2.3 - OpenMP: Step-by-step Example
11
Two examples:● the classic HelloWorld● a matrix multiplication
![Page 13: University of Delaware Dept of Computer & Information Sciences · OpenMP: Environment Variables OpenMP has a set of environment variables that control the runtime execution OMP_NUM_THREADS=num](https://reader035.fdocuments.net/reader035/viewer/2022063012/5fc7a21a32634908943f3e8c/html5/thumbnails/13.jpg)
2.3 (a) - OpenMP: Hello World
12
![Page 14: University of Delaware Dept of Computer & Information Sciences · OpenMP: Environment Variables OpenMP has a set of environment variables that control the runtime execution OMP_NUM_THREADS=num](https://reader035.fdocuments.net/reader035/viewer/2022063012/5fc7a21a32634908943f3e8c/html5/thumbnails/14.jpg)
OpenMP: Environment Variables● OpenMP has a set of environment variables that
control the runtime execution
● OMP_NUM_THREADS=num○ to specify the default number of threads an
OpenMP parallel region should contain● OMP_SCHEDULE=algorithm
○ algorithm = dynamic or static○ the algorithm to be use for scheduling
13
![Page 15: University of Delaware Dept of Computer & Information Sciences · OpenMP: Environment Variables OpenMP has a set of environment variables that control the runtime execution OMP_NUM_THREADS=num](https://reader035.fdocuments.net/reader035/viewer/2022063012/5fc7a21a32634908943f3e8c/html5/thumbnails/15.jpg)
2.3 (a) - OpenMP: Hello World
14
● Compile:○ $> gcc -fopenmp helloworld-omp.c -o helloworld-
omp● Run:
○ $> qlogin -pe threads 8○ $> cd hpc-II○ $> export OMP_NUM_THREADS=8○ $> ./helloworld-omp
![Page 16: University of Delaware Dept of Computer & Information Sciences · OpenMP: Environment Variables OpenMP has a set of environment variables that control the runtime execution OMP_NUM_THREADS=num](https://reader035.fdocuments.net/reader035/viewer/2022063012/5fc7a21a32634908943f3e8c/html5/thumbnails/16.jpg)
2.3 (b) - OpenMP: Matrix Multiply
16
![Page 17: University of Delaware Dept of Computer & Information Sciences · OpenMP: Environment Variables OpenMP has a set of environment variables that control the runtime execution OMP_NUM_THREADS=num](https://reader035.fdocuments.net/reader035/viewer/2022063012/5fc7a21a32634908943f3e8c/html5/thumbnails/17.jpg)
2.3 (b) - OpenMP: Matrix Multiply
17
![Page 18: University of Delaware Dept of Computer & Information Sciences · OpenMP: Environment Variables OpenMP has a set of environment variables that control the runtime execution OMP_NUM_THREADS=num](https://reader035.fdocuments.net/reader035/viewer/2022063012/5fc7a21a32634908943f3e8c/html5/thumbnails/18.jpg)
18
2.3 (b) - OpenMP: Matrix Multiply
![Page 19: University of Delaware Dept of Computer & Information Sciences · OpenMP: Environment Variables OpenMP has a set of environment variables that control the runtime execution OMP_NUM_THREADS=num](https://reader035.fdocuments.net/reader035/viewer/2022063012/5fc7a21a32634908943f3e8c/html5/thumbnails/19.jpg)
19
2.3 (b) - OpenMP: Matrix Multiply
![Page 20: University of Delaware Dept of Computer & Information Sciences · OpenMP: Environment Variables OpenMP has a set of environment variables that control the runtime execution OMP_NUM_THREADS=num](https://reader035.fdocuments.net/reader035/viewer/2022063012/5fc7a21a32634908943f3e8c/html5/thumbnails/20.jpg)
2.3 (b) - OpenMP: Matrix Multiply
20
● #pragma omp parallel shared(A,B,C) private(i,j,k)○ create a parallel region, fork a team of threads (as
many as cores)○ arrays A, B, C are shared among the threads○ the "iterators" are private to each threads
![Page 21: University of Delaware Dept of Computer & Information Sciences · OpenMP: Environment Variables OpenMP has a set of environment variables that control the runtime execution OMP_NUM_THREADS=num](https://reader035.fdocuments.net/reader035/viewer/2022063012/5fc7a21a32634908943f3e8c/html5/thumbnails/21.jpg)
2.3 (b) - OpenMP: Matrix Multiply
21
● #pragma omp for schedule (static)○ the following for-loop have to executed in parallel by
the team○ the schedule clause precise how the iterations have
to be divided■ static/dynamic■ chunk size
![Page 22: University of Delaware Dept of Computer & Information Sciences · OpenMP: Environment Variables OpenMP has a set of environment variables that control the runtime execution OMP_NUM_THREADS=num](https://reader035.fdocuments.net/reader035/viewer/2022063012/5fc7a21a32634908943f3e8c/html5/thumbnails/22.jpg)
2.3 (b) - OpenMP: Matrix Multiply
22
● on Intel i7 4 cores● for 512x512 float matrices● Sequential: 0.92s ● OpenMp : 0.24s
Speedup of 3.83
![Page 23: University of Delaware Dept of Computer & Information Sciences · OpenMP: Environment Variables OpenMP has a set of environment variables that control the runtime execution OMP_NUM_THREADS=num](https://reader035.fdocuments.net/reader035/viewer/2022063012/5fc7a21a32634908943f3e8c/html5/thumbnails/23.jpg)
2.4 - Other Directives and ClausesBut the speedup depends on the input size:
23
![Page 24: University of Delaware Dept of Computer & Information Sciences · OpenMP: Environment Variables OpenMP has a set of environment variables that control the runtime execution OMP_NUM_THREADS=num](https://reader035.fdocuments.net/reader035/viewer/2022063012/5fc7a21a32634908943f3e8c/html5/thumbnails/24.jpg)
24
● Constructs:a. barrier is a synchronisation point for all threads in
the teamb. the block following single will only be executed by
one thread of the teamc. the block following master will only be executed by
the master threadd. only one thread of a team can be in a critical block
at anytimee. sections define an area of the code where
individual section directives delimit independant code to be shared across the threads of the team
2.4 - OpenMP: Construct
![Page 25: University of Delaware Dept of Computer & Information Sciences · OpenMP: Environment Variables OpenMP has a set of environment variables that control the runtime execution OMP_NUM_THREADS=num](https://reader035.fdocuments.net/reader035/viewer/2022063012/5fc7a21a32634908943f3e8c/html5/thumbnails/25.jpg)
25
● clauses:a. shared/private apply to variables listb. default policy for variables sharing
■ either shared or nonec. firstprivate take a list of private variables to be
initializedd. lastprivate take a list of private variables to be
copy oute. reduction take an operation and a list of scalar
variablesf. num_thread either
■ from the team to be used■ in the team
2.4 - OpenMP: Clause
![Page 26: University of Delaware Dept of Computer & Information Sciences · OpenMP: Environment Variables OpenMP has a set of environment variables that control the runtime execution OMP_NUM_THREADS=num](https://reader035.fdocuments.net/reader035/viewer/2022063012/5fc7a21a32634908943f3e8c/html5/thumbnails/26.jpg)
25
2.4 - OpenMP: Barrier example
![Page 27: University of Delaware Dept of Computer & Information Sciences · OpenMP: Environment Variables OpenMP has a set of environment variables that control the runtime execution OMP_NUM_THREADS=num](https://reader035.fdocuments.net/reader035/viewer/2022063012/5fc7a21a32634908943f3e8c/html5/thumbnails/27.jpg)
27
2.4 - OpenMP: Barrier example
![Page 28: University of Delaware Dept of Computer & Information Sciences · OpenMP: Environment Variables OpenMP has a set of environment variables that control the runtime execution OMP_NUM_THREADS=num](https://reader035.fdocuments.net/reader035/viewer/2022063012/5fc7a21a32634908943f3e8c/html5/thumbnails/28.jpg)
2.4 - OpenMP: Reduction
22
![Page 29: University of Delaware Dept of Computer & Information Sciences · OpenMP: Environment Variables OpenMP has a set of environment variables that control the runtime execution OMP_NUM_THREADS=num](https://reader035.fdocuments.net/reader035/viewer/2022063012/5fc7a21a32634908943f3e8c/html5/thumbnails/29.jpg)
2.4 - OpenMP: Construct & Clause
23Source: https://computing.llnl.gov/tutorials/openMP/
![Page 30: University of Delaware Dept of Computer & Information Sciences · OpenMP: Environment Variables OpenMP has a set of environment variables that control the runtime execution OMP_NUM_THREADS=num](https://reader035.fdocuments.net/reader035/viewer/2022063012/5fc7a21a32634908943f3e8c/html5/thumbnails/30.jpg)
2.5 - OpenMP: Q&A
24
![Page 31: University of Delaware Dept of Computer & Information Sciences · OpenMP: Environment Variables OpenMP has a set of environment variables that control the runtime execution OMP_NUM_THREADS=num](https://reader035.fdocuments.net/reader035/viewer/2022063012/5fc7a21a32634908943f3e8c/html5/thumbnails/31.jpg)
3 - MPI
25
● Model ● Language● Step-by-step Example● API● Q&A
![Page 32: University of Delaware Dept of Computer & Information Sciences · OpenMP: Environment Variables OpenMP has a set of environment variables that control the runtime execution OMP_NUM_THREADS=num](https://reader035.fdocuments.net/reader035/viewer/2022063012/5fc7a21a32634908943f3e8c/html5/thumbnails/32.jpg)
3.1 - MPI: Model
26
● Distributed Memory, originally● today implementation support shared memory SMP
Source: https://computing.llnl.gov/tutorials/mpi/
![Page 33: University of Delaware Dept of Computer & Information Sciences · OpenMP: Environment Variables OpenMP has a set of environment variables that control the runtime execution OMP_NUM_THREADS=num](https://reader035.fdocuments.net/reader035/viewer/2022063012/5fc7a21a32634908943f3e8c/html5/thumbnails/33.jpg)
3.2 - MPI: Language
27
● MPI is an Interface○ MPI = Message Passing Interface
● Different implementations are available for C / Fortran
Source: https://computing.llnl.gov/tutorials/mpi/
![Page 34: University of Delaware Dept of Computer & Information Sciences · OpenMP: Environment Variables OpenMP has a set of environment variables that control the runtime execution OMP_NUM_THREADS=num](https://reader035.fdocuments.net/reader035/viewer/2022063012/5fc7a21a32634908943f3e8c/html5/thumbnails/34.jpg)
3.3 - MPI: Step-by-step Example
28Source: https://computing.llnl.gov/tutorials/mpi/
General MPI Program Structure:
![Page 35: University of Delaware Dept of Computer & Information Sciences · OpenMP: Environment Variables OpenMP has a set of environment variables that control the runtime execution OMP_NUM_THREADS=num](https://reader035.fdocuments.net/reader035/viewer/2022063012/5fc7a21a32634908943f3e8c/html5/thumbnails/35.jpg)
3.3 (a) - MPI: Hello World
29
![Page 36: University of Delaware Dept of Computer & Information Sciences · OpenMP: Environment Variables OpenMP has a set of environment variables that control the runtime execution OMP_NUM_THREADS=num](https://reader035.fdocuments.net/reader035/viewer/2022063012/5fc7a21a32634908943f3e8c/html5/thumbnails/36.jpg)
● Compile○ $> mpicc helloworld-mpi.c -o helloworld-mpi○ OR○ $> gcc -c helloworld-mpi.c -o helloworld-mpi.o○ $> mpicc helloworld-mpi.o -o helloworld-mpi○ Warning: Select the good toolchain!
3.3 (a) - MPI: Hello World
30
![Page 37: University of Delaware Dept of Computer & Information Sciences · OpenMP: Environment Variables OpenMP has a set of environment variables that control the runtime execution OMP_NUM_THREADS=num](https://reader035.fdocuments.net/reader035/viewer/2022063012/5fc7a21a32634908943f3e8c/html5/thumbnails/37.jpg)
● Run○ On one node:
■ mpirun -n $NB_PROCCESS ./helloworld-mpi○ On a cluster with qsub (Sun Grid Engine)
■ qsub -pe mpich $NB_PROCESS mpi-qsub.sh■ With mpi-qsub.sh:
3.3 (a) - MPI: Hello World
31
#!/bin/bash##$ -cwd#mpirun -np $NSLOTS ./matmul-mpi
![Page 38: University of Delaware Dept of Computer & Information Sciences · OpenMP: Environment Variables OpenMP has a set of environment variables that control the runtime execution OMP_NUM_THREADS=num](https://reader035.fdocuments.net/reader035/viewer/2022063012/5fc7a21a32634908943f3e8c/html5/thumbnails/38.jpg)
3.3 (b) - MPI: Matrix Multiply
32
![Page 39: University of Delaware Dept of Computer & Information Sciences · OpenMP: Environment Variables OpenMP has a set of environment variables that control the runtime execution OMP_NUM_THREADS=num](https://reader035.fdocuments.net/reader035/viewer/2022063012/5fc7a21a32634908943f3e8c/html5/thumbnails/39.jpg)
3.3 (b) - MPI: Matrix Multiply
33
MPI initialization:
![Page 40: University of Delaware Dept of Computer & Information Sciences · OpenMP: Environment Variables OpenMP has a set of environment variables that control the runtime execution OMP_NUM_THREADS=num](https://reader035.fdocuments.net/reader035/viewer/2022063012/5fc7a21a32634908943f3e8c/html5/thumbnails/40.jpg)
3.3 (b) - MPI: Matrix Multiply
34
Master initialization:
![Page 41: University of Delaware Dept of Computer & Information Sciences · OpenMP: Environment Variables OpenMP has a set of environment variables that control the runtime execution OMP_NUM_THREADS=num](https://reader035.fdocuments.net/reader035/viewer/2022063012/5fc7a21a32634908943f3e8c/html5/thumbnails/41.jpg)
3.3 (b) - MPI: Matrix Multiply
35
![Page 42: University of Delaware Dept of Computer & Information Sciences · OpenMP: Environment Variables OpenMP has a set of environment variables that control the runtime execution OMP_NUM_THREADS=num](https://reader035.fdocuments.net/reader035/viewer/2022063012/5fc7a21a32634908943f3e8c/html5/thumbnails/42.jpg)
3.3 (b) - MPI: Matrix Multiply
36
![Page 43: University of Delaware Dept of Computer & Information Sciences · OpenMP: Environment Variables OpenMP has a set of environment variables that control the runtime execution OMP_NUM_THREADS=num](https://reader035.fdocuments.net/reader035/viewer/2022063012/5fc7a21a32634908943f3e8c/html5/thumbnails/43.jpg)
3.3 (b) - MPI: Matrix Multiply
37
![Page 44: University of Delaware Dept of Computer & Information Sciences · OpenMP: Environment Variables OpenMP has a set of environment variables that control the runtime execution OMP_NUM_THREADS=num](https://reader035.fdocuments.net/reader035/viewer/2022063012/5fc7a21a32634908943f3e8c/html5/thumbnails/44.jpg)
3.4 (b) - MPI API
38
Initialization MPI_Init (&argc,&argv)
Size of the Communicator MPI_Comm_size (comm,&size)
Rank in the Communicator MPI_Comm_rank (comm,&rank)
Terminate all processes in a communicator MPI_Abort (comm,errorcode)
Name of the current processor MPI_Get_processor_name (&name,&resultlength)
Finalize MPI_Finalize ()
Blocking sends MPI_Send(buffer,count,type,dest,tag,comm)
Non-blocking sends MPI_Isend(buffer,count,type,dest,tag,comm,request)
Blocking receive MPI_Recv(buffer,count,type,source,tag,comm,status)
Non-blocking receive MPI_Irecv(buffer,count,type,source,tag,comm,request)
Wait a request MPI_Wait (&request,&status)
Barrier MPI_Barrier (comm)
![Page 45: University of Delaware Dept of Computer & Information Sciences · OpenMP: Environment Variables OpenMP has a set of environment variables that control the runtime execution OMP_NUM_THREADS=num](https://reader035.fdocuments.net/reader035/viewer/2022063012/5fc7a21a32634908943f3e8c/html5/thumbnails/45.jpg)
3.5 - MPI: Q&A
39
![Page 46: University of Delaware Dept of Computer & Information Sciences · OpenMP: Environment Variables OpenMP has a set of environment variables that control the runtime execution OMP_NUM_THREADS=num](https://reader035.fdocuments.net/reader035/viewer/2022063012/5fc7a21a32634908943f3e8c/html5/thumbnails/46.jpg)
5 - Conclusion / Q&A
40
![Page 47: University of Delaware Dept of Computer & Information Sciences · OpenMP: Environment Variables OpenMP has a set of environment variables that control the runtime execution OMP_NUM_THREADS=num](https://reader035.fdocuments.net/reader035/viewer/2022063012/5fc7a21a32634908943f3e8c/html5/thumbnails/47.jpg)
Using Sun Grid Engine● Sun Grid Engine is the queuing system used on
Mills cluster, let see a few command:○ qsub [options] script.qs
■ -pe para_env nbr_slots■ -l
● exclusive=1● standby=1
○ qconf [options]■ -sql : list of all queues■ -sq name : detail of the queue■ -spl : list of parallel environment
○ qstat○ qlogin
15
![Page 48: University of Delaware Dept of Computer & Information Sciences · OpenMP: Environment Variables OpenMP has a set of environment variables that control the runtime execution OMP_NUM_THREADS=num](https://reader035.fdocuments.net/reader035/viewer/2022063012/5fc7a21a32634908943f3e8c/html5/thumbnails/48.jpg)
3.4 (b) - MPI API
30
● MPI_CHAR● MPI_WCHAR● MPI_SHORT● MPI_INT● MPI_LONG● MPI_LONG_LONG_INT● MPI_SIGNED_CHAR● MPI_UNSIGNED_CHAR● MPI_UNSIGNED_SHORT● MPI_UNSIGNED● MPI_UNSIGNED_LONG● MPI_UNSIGNED_LONG_LONG● MPI_FLOAT● MPI_DOUBLE● MPI_LONG_DOUBLE● MPI_C_BOOL● ...
● MPI_Type_contiguous (count, oldtype, &newtype)● MPI_Type_vector (count, blocklength, stride, oldtype, &newtype)● MPI_Type_indexed (count, blocklens[], offsets[], old_type, &newtype)● MPI_Type_commit (&datatype)● MPI_Type_free (&datatype)
● MPI_Bcast (&buffer, count, datatype, root, comm) ● MPI_Scatter (&s_buf, s_cnt, s_type, &r_buf, r_cnt, r_type, root, comm)● MPI_Gather (&s_buf, s_cnt, s_type, &r_buf, r_cnt, r_type, root, comm)● MPI_Reduce (&s_buf, &r_buf, count, datatype, op, root, comm)
○ MPI_MAX○ MPI_MIN○ MPI_SUM○ MPI_PROD○ ...
● MPI_Scan (&s_buf, &r_buf, count, datatype, op, comm)● MPI_Allgather (&s_buf, s_cnt, s_type, &r_buf, r_cnt, r_type, comm)● MPI_Allreduce (&sendbuf,&recvbuf,count,datatype,op,comm) ● MPI_Alltoall (&s_buf, s_cnt, s_type, &r_buf, r_cnt, r_type, comm)