GPU Acceleration at Scale with OpenPower platforms in Code Saturne · GPU Acceleration at Scale...
2
GPU Acceleration at Scale with OpenPower platforms in Code Saturne Samuel Antao IBM Research Daresbury, UK [email protected] Charles Moulinec STFC Daresbury, UK [email protected] Yvan Fournier EDF R&D Paris, France [email protected] Robert Sawko IBM Research Daresbury, UK [email protected] Malgorzata Zimon IBM Research Daresbury, UK [email protected] Christopher Thompson IBM Research Daresbury, UK [email protected] Alex Skillen STFC Daresbury, UK [email protected] Juan Uribe EDF R&D UK Centre Manchester, UK [email protected] David R. Emerson STFC Daresbury, UK [email protected] Abstract—Code Saturne is a widely used computational fluid dynamics software package that uses finite-volume methods to simulate different kinds of flows tailored to tackle multi-bilion-cell unstructured mesh simulations. This class of codes has shown to be challenging to accelerate on GPUs as they consist of many kernels and regular inter-process communication in between. In this poster we show how template pack expansion with CUDA can combine multiple kernels into a single one reducing launching latencies and along with the specification of data environments help reduce host-device communication. We tested these techniques on ORNL Summit Supercomputer based on OpenPOWER platform delivering almost 3x speedup over CPU-only runs on 256 nodes. We also show how the latest generation NVLINK™ interconnect available in POWER9™ improves scaling efficiency, enabling consistent GPU acceleration with just 100K-cells per process. Index Terms—Computational Fluid Dynamics (CFD), GPU, NVLINK, OpenPOWER, Acceleration I. I NTRODUCTION Computational Fluid Dynamics (CFD) has been one of the most important classes of codes targeted by High-performance Computing (HPC) facilities around the world [1]. Code Saturne is one of these codes comprising C/C++, Fortran and Python, with MPI and OpenMP support [2]–[4]. It is an open-source software package developed and maintained by EDF R&D to target very large and complex CFD problems. EDF is one of the leading energy suppliers in Europe, running several nuclear power plants, and Code Saturne has been instrumental to simulate multiple components related with the operation and design of current and next-generation plants, aiming at the simulation of the full nuclear reactors. Therefore, it is key that Code Saturne is able to deliver the required performance at scale for its simulations. The use of accelerators, in particular GPU devices, has become the preferred way to boost application performance as technology does not scale anymore with Moore’s law. GPUs can now be found in many top-tier supercomputers, including the ones based on Open- POWER platforms like Summit (Oak Ridge National Laboratory) OpenPOWER platform. Many CFD applications, in particular the ones that use unstructured discretisations like Code Saturne, have been known to be hard to accelerate on GPUs for the following main reasons: i) unstructured codes map the domain into sparse matrices but GPUs are tailored for dense problems, ii) domain boundaries need to be shared regularly and rank collectives need to be used to compute residuals and control convergence, and iii) complex models have been POWER9 scaling experiments were only possible with Summit Early Access programme sponsored by Oak Ridge Leadership Computing Facil- ity. Work partially funded by UKTC consortium grants EP/L000261/1 and EP/R029326/1. developed over many years and require many kernels scattered in the code base with any kind of dependences between them, making GPU-kernel candidate extraction unfeasible without major code-base disruption. II. PRIOR ART AND GOALS Relatively good solutions to tackle i) have been developed, with many accelerated sparse algebra libraries like cuSparse [5], and therefore it is not a major limiting factor. In our work we use reference implementations of sparse matrix operations and other vector operations [6]. A consequence of ii) is the constant need to move data to/from the device to be shared across ranks as well as frequent barriers for application-wide collectives that preclude streaming. The impact of iii) is translated into latencies for launching many small kernels and communication of small quantities between host and device. Some attempts have been made to accelerate parts of the code like [7] for conjugate gradient or the use of PETSc as an acceleration engine. However, the practicality in real applications is still hindered by ii) and iii), as the problem size per rank/node gets smaller. In this poster we propose a methodology to tackle ii) and iii). To test it we use a 3D- laminar driven cavity solving the velocity using Gauss Seidel (GS) and the pressure an Algebraic Multigrid (AMG), which exposes the problems above with relative low amount of compute. If the compute amount increases the relative impact of the problems above decreases as GPUs can easily accommodate more compute. III. METHODOLOGY We propose to combine three techniques to tackle the problems ii) and iii) mentioned above: a) Use template pack expansion to produce specialised CUDA kernels for any sequence of matrix/vector operations needed in the code: Figure 1 shows how this can be done. any_kernels is the only device entry point (CUDA kernel). This templated CUDA kernel takes a list of enumerators Kinds, each specifying a vector/matrix operation. It also takes an array KernelArgsSeriesGPU in CUDA constant memory with the arguments for each of these operations. The template pack is then expanded along with the right indexing to the array of arguments, and any_kernel is called for each operation. The function any_kernel contains a switch case statement where each case is a vector/matrix operation. The switch is controlled by the template argument Kind, hence each instance of any_kernel is optimised into a single vector/matrix operation. The result is a given instance of any_kernels to be a specialised sequence of vector/matrix operations without any launching latencies
Transcript of GPU Acceleration at Scale with OpenPower platforms in Code Saturne · GPU Acceleration at Scale...
GPU Acceleration at Scale with OpenPower platforms inCode Saturne
Samuel AntaoIBM ResearchDaresbury, UK
Charles MoulinecSTFC
Daresbury, [email protected]
Yvan FournierEDF R&D
Paris, [email protected]
Robert SawkoIBM ResearchDaresbury, UK
Malgorzata ZimonIBM ResearchDaresbury, UK
Christopher ThompsonIBM ResearchDaresbury, UK
Alex SkillenSTFC
Daresbury, [email protected]
Juan UribeEDF R&D UK Centre
Manchester, [email protected]
David R. EmersonSTFC
Daresbury, [email protected]
Abstract—Code Saturne is a widely used computational fluid dynamicssoftware package that uses finite-volume methods to simulate differentkinds of flows tailored to tackle multi-bilion-cell unstructured meshsimulations. This class of codes has shown to be challenging to accelerateon GPUs as they consist of many kernels and regular inter-processcommunication in between. In this poster we show how template packexpansion with CUDA can combine multiple kernels into a single onereducing launching latencies and along with the specification of dataenvironments help reduce host-device communication. We tested thesetechniques on ORNL Summit Supercomputer based on OpenPOWERplatform delivering almost 3x speedup over CPU-only runs on 256nodes. We also show how the latest generation NVLINK™ interconnectavailable in POWER9™ improves scaling efficiency, enabling consistentGPU acceleration with just 100K-cells per process.
Index Terms—Computational Fluid Dynamics (CFD), GPU, NVLINK,OpenPOWER, Acceleration
I. INTRODUCTION
Computational Fluid Dynamics (CFD) has been one of the mostimportant classes of codes targeted by High-performance Computing(HPC) facilities around the world [1]. Code Saturne is one of thesecodes comprising C/C++, Fortran and Python, with MPI and OpenMPsupport [2]–[4]. It is an open-source software package developedand maintained by EDF R&D to target very large and complexCFD problems. EDF is one of the leading energy suppliers inEurope, running several nuclear power plants, and Code Saturne hasbeen instrumental to simulate multiple components related with theoperation and design of current and next-generation plants, aiming atthe simulation of the full nuclear reactors. Therefore, it is key thatCode Saturne is able to deliver the required performance at scale forits simulations.
The use of accelerators, in particular GPU devices, has becomethe preferred way to boost application performance as technologydoes not scale anymore with Moore’s law. GPUs can now be foundin many top-tier supercomputers, including the ones based on Open-POWER platforms like Summit (Oak Ridge National Laboratory)OpenPOWER platform. Many CFD applications, in particular theones that use unstructured discretisations like Code Saturne, havebeen known to be hard to accelerate on GPUs for the following mainreasons: i) unstructured codes map the domain into sparse matricesbut GPUs are tailored for dense problems, ii) domain boundaries needto be shared regularly and rank collectives need to be used to computeresiduals and control convergence, and iii) complex models have been
POWER9 scaling experiments were only possible with Summit EarlyAccess programme sponsored by Oak Ridge Leadership Computing Facil-ity. Work partially funded by UKTC consortium grants EP/L000261/1 andEP/R029326/1.
developed over many years and require many kernels scattered inthe code base with any kind of dependences between them, makingGPU-kernel candidate extraction unfeasible without major code-basedisruption.
II. PRIOR ART AND GOALS
Relatively good solutions to tackle i) have been developed, withmany accelerated sparse algebra libraries like cuSparse [5], andtherefore it is not a major limiting factor. In our work we usereference implementations of sparse matrix operations and othervector operations [6].
A consequence of ii) is the constant need to move data to/fromthe device to be shared across ranks as well as frequent barriersfor application-wide collectives that preclude streaming. The impactof iii) is translated into latencies for launching many small kernels andcommunication of small quantities between host and device. Someattempts have been made to accelerate parts of the code like [7] forconjugate gradient or the use of PETSc as an acceleration engine.However, the practicality in real applications is still hindered by ii)and iii), as the problem size per rank/node gets smaller. In this posterwe propose a methodology to tackle ii) and iii). To test it we use a 3D-laminar driven cavity solving the velocity using Gauss Seidel (GS)and the pressure an Algebraic Multigrid (AMG), which exposes theproblems above with relative low amount of compute. If the computeamount increases the relative impact of the problems above decreasesas GPUs can easily accommodate more compute.
III. METHODOLOGY
We propose to combine three techniques to tackle the problems ii)and iii) mentioned above:
a) Use template pack expansion to produce specialised CUDAkernels for any sequence of matrix/vector operations needed in thecode: Figure 1 shows how this can be done. any_kernels is theonly device entry point (CUDA kernel). This templated CUDA kerneltakes a list of enumerators Kinds, each specifying a vector/matrixoperation. It also takes an array KernelArgsSeriesGPU inCUDA constant memory with the arguments for each of theseoperations. The template pack is then expanded along with the rightindexing to the array of arguments, and any_kernel is called foreach operation. The function any_kernel contains a switch casestatement where each case is a vector/matrix operation. The switchis controlled by the template argument Kind, hence each instanceof any_kernel is optimised into a single vector/matrix operation.The result is a given instance of any_kernels to be a specialisedsequence of vector/matrix operations without any launching latencies
between them. We also control the amount of rows per block, sothat each block always produce results for the same rows and, withthis, avoid inter-block synchronisations. The outcome is similar totask abstractions found in programming models like OpenMP [8],but with tasks handled/queued at compile time with CUDA.
IBM Confidential GPU acceleration in Code Saturne
1 template <KernelKinds Kind >
2 __device__ int any_kernel(KernelArgsBase &Arg , unsigned n_rows_per_block) {
3 switch(Kind) {
4 /* ... */
5 // Dot product:
6 //
7 case DP_xx:
8 dot_product <Kind >(
9 /* version */ Arg.getArg <cs_lnum_t >(0),
10 /* n_rows */ Arg.getArg <cs_lnum_t >(1),
11 /* x */ Arg.getArg <cs_real_t *>(2),
12 /* y */ nullptr ,
13 /* z */ nullptr ,
14 /* res */ Arg.getArg <cs_real_t *>(3),
15 /* n_rows_per_block */ n_rows_per_block);
16 break;
17 /* ... */
18 }
19 __syncthreads ();
20 return 0;
21 }
22
23 template <KernelKinds ... Kinds >
24 __global__ void any_kernels(void) {
25
26 auto *KA = reinterpret_cast <KernelArgsSeries *>(& KernelArgsSeriesGPU [0]);
27 const unsigned n_rows_per_block = KA ->RowsPerBlock;
28 unsigned idx = 0;
29
30 int dummy [] = { any_kernel <Kinds >(KA ->Args[idx++], n_rows_per_block)... };
31 (void) dummy;
32 }
Listing 10: Device entry-point function for kernel execution.
and free by the appropriate CUDA runtime call. However, in Code Saturne, the storage data is copiedfrom is in many cases allocated in the Fortran part of the code. As there is no Fortran equivalent for thepinned memory allocation, we had to rely on ISO C bindings to implement that and replace calls to theallocate and delete intrinsics - see Listing 11.
3.5 Results
We applied the changes described in Section 3.4 to all solver code used by the selected testcase, includingthe Gauss-Seidel solver. Figure 2 presents the execution time, with and without GPU acceleration, aswell as the speedup, for a single-rank, single-core run for di↵erent numbers of host OpenMP threads. Thenumber of CUDA threads per block is 128 and the number of blocks is such that each thread is assigned8 iterations out of the total iteration space (number of rows).
The time di↵erence between solver and wall time is consistent between runs with and without GPUacceleration, meaning that the GPU acceleration does not cause noticeable performance loss in otherparts of the program. We see a sub-linear performance improvement when using more OpenMP threadsfor CPU-only runs. Wall time for the GPU also benefits from more host threading as there are sectionsof the code that use OpenMP threading and are not GPU accelerated. The solvers time is improved bymore than 5x in the GPU accelerated code, but the wall clock is only over 3x as the solvers executiontime starts to account for about half of the wall time. This is corroborated by the profiler breakdownobtained for the GPU accelerated code:
• 47.1% - Solvers total time
– 8.9% - Gauss-Seidel solver with MSR matrix format
– 37.5% - Algebraic multigrid solver
⇤ 16.5% - Multigrid setup
13
Fig. 1: Template pack expansion to combine multiple operations intoa single CUDA kernel.
b) Manage device memory explicitly: When the applicationstarts, a pool of memory is allocated and reused for all data that needsto be on the device. As the amount of variables live on the deviceis small, the pool management is lightweight. The implementationcan with this combine arguments for consecutive kernels and createdata environments so that data stays on the device between series ofoperations, reducing latencies associated with data movements.
c) Use NVIDIA Multi-Process Service to increase GPU utilisa-tion by having multiple ranks using the same GPU: As there are stilltasks that need to be handled by the CPU, (e.g. ghost cell samplingprior to communication with other ranks) and data to be movedto/from the device, the GPU could sometimes be idle. Overlappingexecution of multiple ranks in the GPU allows the GPU-compute ofa given rank to overlap data movements or CPU-compute of someother rank, improving utilisation and efficiency.
IV. RESULTS
We tested our methodology with the configuration in Table I.Figure 2 shows how the proposed techniques improve performancefor our target testcase which couples GS and AMG algorithms. Fig-ure 3 compares CPU-only and CPU+GPU runs. Consistent speedupis observed (over 2x) even for mesh partitions as small as 100K-cell per rank. Even better scaling performance are shown usingPOWER9 versus POWER8 machines - 16x more resources can beefficiently used for a problem that is only 8x larger. NVLINK 2.0(POWER9) plays an important role as it almost halves data movementlatencies comparing to NVLINK 1.0 (POWER8). Efficiencies higherthan 100% can explained with more advantageous domain partitionsfor that specific number of ranks.
Cores Ranks GPUs Ranks/GPU OpenMP thd./rankPOWER8 only 20 20 4 - 8POWER8 + GPU 5POWER9 only 42 21 6 - 4POWER9 +GPU 18 3
TABLE I: Configuration per node for both target systems. The chosensettings are the ones delivering better performance.
V. CONCLUSIONS
In this poster we propose a set of techniques to improve GPUutilisation in Code Saturne by aggressively reducing idle times anddata movement latencies. We combine template pack expansion,explicit device memory pool management and the NVIDIA Multi-Process Service to enable over 2x speedup over CPU-only runseven when using only 100K-cell per rank, improving strong scalingCode Saturne performance and enabling it to tackle larger problemsat scale. All the techniques proposed in this poster can be applied
IBM Confidential GPU acceleration in Code Saturne
(a) AMG cycle overview.
(b) Coarse mesh iteration detail.
Figure 1: OpenMP 4.5 profiling overview of the AMG solver.
3.4 CUDA porting
We implemented a CUDA porting providing the same functionality as the OpenMP 4.5 porting, whileaddressing the limitations listed in Section 3.3. We implemented the port in a separate folder using C++and exposed a C interface so that it could be invoked from within Code Saturne existing code.
3.4.1 Kernels’ implementation
We implemented four templated kernels using the CUDA language: Gauss-Seidel, SpMV, dot productsand stream operations. The di↵erent variations in SpMV (MSR, CSR, with/without diagonal) are con-trolled with template arguments. Similarly, we use template arguments to reuse code for the di↵erentflavours of dot product and other stream operations. For the Gauss-Seidel kernel, we use a templateargument to control whether the kernel is combined with the computation of a dot-product. Using tem-plated functions improves the code scalability without increasing the complexity of control flows duringthe execution. Listings 6 and 7 present the implementation of the SpMV kernel and stream operations.
The implementation of SpMV (Gauss-Seidel kernels are very similar) is similar to the OpenMP im-plementation except for the reductions, where we can benefit from the GPU double-precision shu✏einstructions which deliver better performance than communicating through the scratchpad memory. Theimplementation of the dot products also benefit from the shu✏e operation to perform reductions acrossthe warp.
3.4.2 Memory management
We implemented a complete infrastructure to manage the GPU storage and the mapping to host variables.During the program initialisation phase, the implementation detects the number of ranks using a givenGPU and chunks all the available GPU memory by rank. Then, each rank manages its own chunk,without any expensive memory allocation/deallocation, and making sure that all variables are 256-bytealigned (GPU cache block-size). When a given variable needs to be allocated in GPU memory, theimplementation detects the minimum slot that fits the requested size in the already allocated memory.Then, the map between the host address and the device address is kept in a list so that it can be retrievedlater. As the number of variables mapped to the GPU at a given point is small, managing the list of
8
(a) Original AMG single-rank timeline.
IBM Confidential GPU acceleration in Code Saturne
(a) Gauss-Seidel solver.
(b) AMG solver finer mesh.
(c) AMG solver coarser mesh.
Figure 3: Single-rank timeline details for the CUDA port using a 1.5M-cell mesh.
⇤ 8.3% - Conjugate gradient - includes MSR and CST SpMVs.
⇤ 7.0% - Compute coarse cells from fine cells
We observe that the number of iterations for the Gauss-Seidel solver increases from 64 to 76 for theGPU accelerated version. This solver relies on a data race during the execution of the solver loop in thesense that the input vector is also the output vector. Therefore, is possible that the output of an iterationis used as input in some other iteration, depending on how the di↵erent threads are scheduled. This datasharing between iterations contributes to the solver to converge with less iterations. For the GPU versionof the code, the number of active threads in the execution of the loop is two or more orders of magnitudehigher than the active threads in the CPU version (only up to 8). This causes the propagation of databetween iterations less likely to happen, and therefore the solver will take longer to converge as it becomesmore similar to a Jacobi solver rather than a Gauss-Seidel solver.
Figure 3 shows the detail of the GPU execution timeline for the Gauss-Seidel solver, and the coarserand finer levels of the AMG solver. We observe that the Gauss-Seidel and the AMG finer mesh solversmake a better use of the GPU. The AMG finer mesh solver still accounts a significant amount of idleGPU time. A methodology to reduce the amount of idle time is discussed in section 4.
15
(b) AMG single-rank timeline with template pack expansion and dataenvironments.
IBM Confidential GPU acceleration in Code Saturne
(a) Gauss-Seidel solver.
(b) AMG solver finer mesh.
(c) AMG solver coarser mesh.
Figure 6: Details for the GPU execution of 5 ranks bound to a single GPU.
19
(c) AMG timeline with MPS with 5 ranks per GPU.
IBM Confidential GPU acceleration in Code Saturne
(a) Gauss-Seidel solver.
(b) AMG solver finer mesh.
(c) AMG solver coarser mesh.
Figure 6: Details for the GPU execution of 5 ranks bound to a single GPU.
19
(d) GS timeline with MPS with 5 ranks per GPU.
Fig. 2: Execution timelines on POWER8+GPU for AMG and GS.(2a) Large latencies between compute kernels and data movements.(2b) Idle times are greatly reduced. (2c) Data movements is hidden byoverlapping multiple ranks. (2d) Data movements almost completelyhidden for GS.
2.39 2.422.32
2.222.08
2.00
100.0% 98.1% 92.9% 83.1%
71.2% 65.0%
0%
20%
40%
60%
80%
100%
120%
1.4
1.6
1.8
2.0
2.2
2.4
2.6
2.8
3.0
1 2 4 8 16 32
Efficiency
SpeedupoverCPU
-only(1x)
NodesSpeedup Efficiency
POWER8 + P100 GPUs (111M-cells).
2.31
2.90
2.34
100.0% 111.4%
84.8%
0%
20%
40%
60%
80%
100%
120%
1.5
1.7
1.9
2.1
2.3
2.5
2.7
2.9
3.1
3.3
3.5
64 256 512
Efficiency
SpeedupoverCPU
-only(1x)
NodesSpeedup Efficiency
POWER9 + V100 GPUs (889M-cells).1
Fig. 3: CPU+GPU speedup over CPU-only and efficiency (strong).Speedup is reported on same number of nodes.
to other codes whose acceleration at scale is limited by latenciesresulting from running multiple small kernels.
REFERENCES
[1] A. Turner, “Parallel software usage on UK national HPC facilities 2009-2015: How well have applications kept up with increasingly parallelhardware?” EPCC, Tech. Rep., 04 2015.
[2] EDF. (2018) Code saturne official web page. [Online]. Available:https://www.code-saturne.org
[3] F. Archambeau, N. Mechitoua, and M. Sakiz, “Code saturne: a finitevolume code for the computation of turbulent incompressible flows,”International Journal on Finite Volumes, vol. 1, 2004.
[4] Y. Fournier, J. Bonelle, C. Moulinec, Z. Shang, A. Sunderland, andJ. Uribe, “Optimizing code saturne computations on petascale systems,”Computers & Fluids, vol. 45, no. 1, pp. 103 – 108, 2011, 22nd Int.Conference on Parallel Computational Fluid Dynamics (ParCFD 2010).
[5] NVIDIA. (2018) cuSparse official documentation web page. [Online].Available: https://docs.nvidia.com/cuda/cusparse/
[6] N. Bell and M. Garland, “Efficient sparse matrix-vector multiplication onCUDA,” NVIDIA, Tech. Rep., 11 2008.
[7] H. Anzt, M. Baboulin, J. Dongarra, Y. Fournier, F. Hulsemann,A. Khabou, and Y. Wang, “Accelerating the conjugate gradient algorithmwith GPUs in CFD simulations,” in High Performance Computing forComputational Science – VECPAR 2016, 2017, pp. 35–43.
[8] OpenMP Architecture Review Board, “Openmp application programminginterface version 4.5,” OpenMP ARB, Tech. Rep., 11 2015.
