GPU Acceleration of Unmodified CSM and CFD Solvers · Multiplicative vertically (between levels),...
Transcript of GPU Acceleration of Unmodified CSM and CFD Solvers · Multiplicative vertically (between levels),...
![Page 1: GPU Acceleration of Unmodified CSM and CFD Solvers · Multiplicative vertically (between levels), global coarse grid problem (MG-like) Additive horizontally: block-Jacobi / Schwarz](https://reader035.fdocuments.net/reader035/viewer/2022071011/5fc9513df88d897bf1617d9c/html5/thumbnails/1.jpg)
GPU Acceleration of Unmodified
CSM and CFD Solvers
Dominik GöddekeSven H.M. Buijssen, Hilmar Wobker and Stefan Turek
Angewandte Mathematik und NumerikTU Dortmund, Germany
High Performance Computing & SimulationWorkshop on Architecture-aware Simulation and Computing
Leipzig, June 22, 2009
Introduction FEAST Co-processor integration Results Conclusions
![Page 2: GPU Acceleration of Unmodified CSM and CFD Solvers · Multiplicative vertically (between levels), global coarse grid problem (MG-like) Additive horizontally: block-Jacobi / Schwarz](https://reader035.fdocuments.net/reader035/viewer/2022071011/5fc9513df88d897bf1617d9c/html5/thumbnails/2.jpg)
The big picture
Scientific computing is in the middle of a paradigm shift
ILP wall memory wall characteristic feature size
heat power consumption leaking voltage
Hardware evolves towards parallelism and heterogeneity
multicore CPUs Cell BE processor GPUs
Emerging manycore architectures
accelerators algorithm design for 10000s of threads
Introduction FEAST Co-processor integration Results Conclusions
![Page 3: GPU Acceleration of Unmodified CSM and CFD Solvers · Multiplicative vertically (between levels), global coarse grid problem (MG-like) Additive horizontally: block-Jacobi / Schwarz](https://reader035.fdocuments.net/reader035/viewer/2022071011/5fc9513df88d897bf1617d9c/html5/thumbnails/3.jpg)
FEAST –
Hardware-oriented Numerics
Introduction FEAST Co-processor integration Results Conclusions
![Page 4: GPU Acceleration of Unmodified CSM and CFD Solvers · Multiplicative vertically (between levels), global coarse grid problem (MG-like) Additive horizontally: block-Jacobi / Schwarz](https://reader035.fdocuments.net/reader035/viewer/2022071011/5fc9513df88d897bf1617d9c/html5/thumbnails/4.jpg)
Mesh structure
Fully adaptive gridsMaximum flexibility‘Stochastic’ numberingUnstructured sparse matricesIndirect addressing, very slow.
Locally structured gridsLogical tensor productFixed banded matrix structureDirect addressing (⇒ fast)r -adaptivity
Unstructured macro mesh of tensor product subdomains
UU
“window” formatrix-vectormultiplication,per macro
hierarchicallyrefined subdomain(= “macro”),rowwise numbered
unstructured mesh
UDUL
DUDDDL
LULDLL
I-1
I
I+1
I-M-1I-M
I-M+1I+M-1
I+M
I+M+1
Ωi
Introduction FEAST Co-processor integration Results Conclusions
![Page 5: GPU Acceleration of Unmodified CSM and CFD Solvers · Multiplicative vertically (between levels), global coarse grid problem (MG-like) Additive horizontally: block-Jacobi / Schwarz](https://reader035.fdocuments.net/reader035/viewer/2022071011/5fc9513df88d897bf1617d9c/html5/thumbnails/5.jpg)
Solver structure
ScaRC – Scalable Recursive Clustering
Minimal overlap by extended Dirichlet BCs
Hybrid multilevel domain decomposition method
Inspired by parallel MG (”best of both worlds”)
Multiplicative vertically (between levels), global coarse grid problem(MG-like)Additive horizontally: block-Jacobi / Schwarz smoother (DD-like)
Hide local irregularities by MGs within the Schwarz smoother
Embed in Krylov to alleviate Block-Jacobi character
global BiCGStab
preconditioned by
global multilevel (V 1+1)
additively smoothed by
for all Ωi : local multigrid
coarse grid solver: UMFPACK
Introduction FEAST Co-processor integration Results Conclusions
![Page 6: GPU Acceleration of Unmodified CSM and CFD Solvers · Multiplicative vertically (between levels), global coarse grid problem (MG-like) Additive horizontally: block-Jacobi / Schwarz](https://reader035.fdocuments.net/reader035/viewer/2022071011/5fc9513df88d897bf1617d9c/html5/thumbnails/6.jpg)
Multivariate problems
Block-structured systems
Guiding idea: Tune scalar case once per architecture instead of overand over again per application
Equation-wise ordering of the unknowns
Block-wise treatment enables multivariate ScaRC solvers
Examples
Linearised elasticity with compressible material (2x2 blocks)
Saddle point problems: Stokes, linearised elasticity with (nearly)incompressible material, Navier-Stokes with stabilisation (3x3 blocks,three zero Blocks for Stokes)
(
A11 A12
A21 A22
)(
u1
u2
)
= f,
A11 0 B1
0 A22 B2
BT
1BT
20
v1
v2
p
= f,
A11 and A22 correspond to scalar (elliptic) operators⇒ Tuned linear algebra and tuned solvers
Introduction FEAST Co-processor integration Results Conclusions
![Page 7: GPU Acceleration of Unmodified CSM and CFD Solvers · Multiplicative vertically (between levels), global coarse grid problem (MG-like) Additive horizontally: block-Jacobi / Schwarz](https://reader035.fdocuments.net/reader035/viewer/2022071011/5fc9513df88d897bf1617d9c/html5/thumbnails/7.jpg)
Co-processor integrationinto FEAST
Introduction FEAST Co-processor integration Results Conclusions
![Page 8: GPU Acceleration of Unmodified CSM and CFD Solvers · Multiplicative vertically (between levels), global coarse grid problem (MG-like) Additive horizontally: block-Jacobi / Schwarz](https://reader035.fdocuments.net/reader035/viewer/2022071011/5fc9513df88d897bf1617d9c/html5/thumbnails/8.jpg)
Bandwidth in a CPU/GPU node
Introduction FEAST Co-processor integration Results Conclusions
![Page 9: GPU Acceleration of Unmodified CSM and CFD Solvers · Multiplicative vertically (between levels), global coarse grid problem (MG-like) Additive horizontally: block-Jacobi / Schwarz](https://reader035.fdocuments.net/reader035/viewer/2022071011/5fc9513df88d897bf1617d9c/html5/thumbnails/9.jpg)
Mixed Precision Multigrid
Core2Duo (double) GTX 280 (mixed)Level time(s) MFLOP/s time(s) MFLOP/s speedup
7 0.021 1405 0.009 2788 2.3x8 0.094 1114 0.012 8086 7.8x9 0.453 886 0.026 15179 17.4x10 1.962 805 0.073 21406 26.9x
Poisson on unitsquare, Dirichlet BCs, TP grid, not a matrix stencil
1M DOF, multigrid, FE-accurate in less than 0.1 seconds!
Converges against wrong solution in single precision
27x faster than CPU, exactly same results as pure double
1.7x faster than pure double on GPU
8800 GTX (correction loop on CPU): 0.44 seconds on level 10
Introduction FEAST Co-processor integration Results Conclusions
![Page 10: GPU Acceleration of Unmodified CSM and CFD Solvers · Multiplicative vertically (between levels), global coarse grid problem (MG-like) Additive horizontally: block-Jacobi / Schwarz](https://reader035.fdocuments.net/reader035/viewer/2022071011/5fc9513df88d897bf1617d9c/html5/thumbnails/10.jpg)
Minimally invasive integration
global BiCGStab
preconditioned by
global multilevel (V 1+1)
additively smoothed by
for all Ωi : local multigrid
coarse grid solver: UMFPACK
All outer work: CPU, double
Local MGs: GPU, single
GPU performs preconditioning
Applicable to many co-processors
Introduction FEAST Co-processor integration Results Conclusions
![Page 11: GPU Acceleration of Unmodified CSM and CFD Solvers · Multiplicative vertically (between levels), global coarse grid problem (MG-like) Additive horizontally: block-Jacobi / Schwarz](https://reader035.fdocuments.net/reader035/viewer/2022071011/5fc9513df88d897bf1617d9c/html5/thumbnails/11.jpg)
Minimally invasive integration
General approach
Balance acceleration potential and integration effort
Accelerate many different applications built on top of one central FEand solver toolkit
Diverge code paths as late as possible
No changes to application code!
Retain all functionality
Do not sacrifice accuracy
Challenges
Heterogeneous task assignment to maximise throughput
Limited device memory (modeled as huge L3 cache)
Overlapping CPU and GPU computations
Building dense accelerated clusters
Introduction FEAST Co-processor integration Results Conclusions
![Page 12: GPU Acceleration of Unmodified CSM and CFD Solvers · Multiplicative vertically (between levels), global coarse grid problem (MG-like) Additive horizontally: block-Jacobi / Schwarz](https://reader035.fdocuments.net/reader035/viewer/2022071011/5fc9513df88d897bf1617d9c/html5/thumbnails/12.jpg)
Some results
Introduction FEAST Co-processor integration Results Conclusions
![Page 13: GPU Acceleration of Unmodified CSM and CFD Solvers · Multiplicative vertically (between levels), global coarse grid problem (MG-like) Additive horizontally: block-Jacobi / Schwarz](https://reader035.fdocuments.net/reader035/viewer/2022071011/5fc9513df88d897bf1617d9c/html5/thumbnails/13.jpg)
Linearised elasticity
(
A11 A12
A21 A22
)(
u1
u2
)
= f
(
(2µ +λ )∂xx + µ∂yy (µ +λ )∂xy
(µ +λ )∂yx µ∂xx +(2µ +λ )∂yy
)
global multivariate BiCGStabblock-preconditioned by
Global multivariate multilevel (V 1+1)additively smoothed (block GS) by
for all Ωi : solve A11c1 = d1 bylocal scalar multigrid
update RHS: d2 = d2 −A21c1
for all Ωi : solve A22c2 = d2 bylocal scalar multigrid
coarse grid solver: UMFPACK
Introduction FEAST Co-processor integration Results Conclusions
![Page 14: GPU Acceleration of Unmodified CSM and CFD Solvers · Multiplicative vertically (between levels), global coarse grid problem (MG-like) Additive horizontally: block-Jacobi / Schwarz](https://reader035.fdocuments.net/reader035/viewer/2022071011/5fc9513df88d897bf1617d9c/html5/thumbnails/14.jpg)
Accuracy (I)
1e-8
1e-7
1e-6
1e-5
1e-4
16 64 256
<--
-- s
mal
ler
is b
ette
r <
----
L2
erro
r
number of subdomains
L7(CPU)L7(GPU)L8(CPU)L8(GPU)L9(CPU)L9(GPU)
L10(CPU)L10(GPU)
Same results for CPU and GPU
L2 error against analytically prescribed displacements
Tests on 32 nodes, 512M DOF
Introduction FEAST Co-processor integration Results Conclusions
![Page 15: GPU Acceleration of Unmodified CSM and CFD Solvers · Multiplicative vertically (between levels), global coarse grid problem (MG-like) Additive horizontally: block-Jacobi / Schwarz](https://reader035.fdocuments.net/reader035/viewer/2022071011/5fc9513df88d897bf1617d9c/html5/thumbnails/15.jpg)
Accuracy (II)
Cantilever beam, aniso 1:1, 1:4, 1:16Hard, very ill-conditioned CSM testCG solver: > 2x iterations per refinementGPU-ScaRC solver: same results as CPU
128
256
512
1024
2048
4096
8192
16384
32768
65536
2.1Ki 8.4Ki 33.2Ki 132Ki 528Ki 2.1Mi 8.4Mi
<--
-- s
maller
is b
etter
<--
--num
ber
of itera
tions
number of DOF
aniso01aniso04aniso16
aniso04 Iterations Volume y-Displacementrefinement L CPU GPU CPU GPU CPU GPU
8 4 4 1.6087641E-3 1.6087641E-3 -2.8083499E-3 -2.8083499E-39 4 4 1.6087641E-3 1.6087641E-3 -2.8083628E-3 -2.8083628E-310 4.5 4.5 1.6087641E-3 1.6087641E-3 -2.8083667E-3 -2.8083667E-3
aniso16
8 6 6 6.7176398E-3 6.7176398E-3 -6.6216232E-2 -6.6216232E-29 6 5.5 6.7176427E-3 6.7176427E-3 -6.6216551E-2 -6.6216552E-210 5.5 5.5 6.7176516E-3 6.7176516E-3 -6.6217501E-2 -6.6217502E-2
Introduction FEAST Co-processor integration Results Conclusions
![Page 16: GPU Acceleration of Unmodified CSM and CFD Solvers · Multiplicative vertically (between levels), global coarse grid problem (MG-like) Additive horizontally: block-Jacobi / Schwarz](https://reader035.fdocuments.net/reader035/viewer/2022071011/5fc9513df88d897bf1617d9c/html5/thumbnails/16.jpg)
Weak scalability
0
20
40
60
80
100
120
140
160
8 16 32 64 96 128 160
<--
-- s
mal
ler
is b
ette
r <
----
sec
Number of nodes
2c8m_Cn_L10_const1g8m_Cn_L10_const1c8m_Cn_L10_const
20
25
30
35
40
45
50
4 8 16 32 64 128
<--
-- s
mal
ler
is b
ette
r <
----
nor
mal
ized
tim
e pe
r ite
ratio
n in
sec
Number of nodes
CPUGPU
Outdated cluster, dual Xeon EM64T,
one NVIDIA Quadro FX 1400 per node (one generation behind theXeons, 20GB/s BW)
Poisson problem (left): up to 1.3B DOF, 160 nodes
Elasticity (right): up to 1B DOF, 128 nodes
Introduction FEAST Co-processor integration Results Conclusions
![Page 17: GPU Acceleration of Unmodified CSM and CFD Solvers · Multiplicative vertically (between levels), global coarse grid problem (MG-like) Additive horizontally: block-Jacobi / Schwarz](https://reader035.fdocuments.net/reader035/viewer/2022071011/5fc9513df88d897bf1617d9c/html5/thumbnails/17.jpg)
Absolute speedup
0
50
100
150
200
250
BLOCK CRACK PIPE STEELFRAME
<--
-- s
mal
ler
is b
ette
r <
----
time
(sec
)
CPU-singleCPU-dual
GPU-single
16 nodes, Opteron X2 2214,
NVIDIA Quadro FX 5600 (76GB/s BW), OpenGL
Problem size 128M DOF
Dualcore 1.6x faster than singlecore
GPU 2.6x faster than singlecore, 1.6x than dual
Introduction FEAST Co-processor integration Results Conclusions
![Page 18: GPU Acceleration of Unmodified CSM and CFD Solvers · Multiplicative vertically (between levels), global coarse grid problem (MG-like) Additive horizontally: block-Jacobi / Schwarz](https://reader035.fdocuments.net/reader035/viewer/2022071011/5fc9513df88d897bf1617d9c/html5/thumbnails/18.jpg)
Acceleration analysis
Speedup analysis
Addition of GPUs increases resources
⇒ Correct model: strong scalability inside each node
Accelerable fraction of the elasticity solver: 2/3
Remaining time spent in MPI and the outer solver
Accelerable fraction Racc: 66%Local speedup Slocal: 9xTotal speedup Stotal: 2.6xTheoretical limit Smax: 3x
1
1.5
2
2.5
3
3.5
4
1 2 3 4 5 6 7 8 9 10
----
> la
rger
is b
ette
r --
-->
Sto
tal
Slocal
GPUX
SSE
X
Racc = 1/2Racc = 2/3Racc = 3/4
Introduction FEAST Co-processor integration Results Conclusions
![Page 19: GPU Acceleration of Unmodified CSM and CFD Solvers · Multiplicative vertically (between levels), global coarse grid problem (MG-like) Additive horizontally: block-Jacobi / Schwarz](https://reader035.fdocuments.net/reader035/viewer/2022071011/5fc9513df88d897bf1617d9c/html5/thumbnails/19.jpg)
Stationary Navier-Stokes
A11 A12 B1
A21 A22 B2
BT
1BT
2C
u1
u2
p
=
f1f2g
4-node cluster
Opteron X2 2214
GeForce 8800 GTX(90GB/s BW), CUDA
Driven cavity and channelflow around a cylinder
fixed point iterationsolving linearised subproblems with
global BiCGStab (reduce initial residual by 1 digit)Block-Schurcomplement preconditioner1) approx. solve for velocities with
global MG (V 1+0), additively smoothed by
for all Ωi : solve for u1 withlocal MG
for all Ωi : solve for u2 withlocal MG
2) update RHS: d3 = −d3 +BT(c1,c2)T
3) scale c3 = (MLp )−1d3
Introduction FEAST Co-processor integration Results Conclusions
![Page 20: GPU Acceleration of Unmodified CSM and CFD Solvers · Multiplicative vertically (between levels), global coarse grid problem (MG-like) Additive horizontally: block-Jacobi / Schwarz](https://reader035.fdocuments.net/reader035/viewer/2022071011/5fc9513df88d897bf1617d9c/html5/thumbnails/20.jpg)
Navier-Stokes results
Speedup analysis
Racc Slocal Stotal
L9 L10 L9 L10 L9 L10DC Re100 41% 46% 6x 12x 1.4x 1.8xDC Re250 56% 58% 5.5x 11.5x 1.9x 2.1xChannel flow 60% – 6x – 1.9x –
Important consequence: Ratio between assembly and linear solvechanges significantly
DC Re100 DC Re250 Channel flow
plain accel. plain accel. plain accel.29:71 50:48 11:89 25:75 13:87 26:74
Introduction FEAST Co-processor integration Results Conclusions
![Page 21: GPU Acceleration of Unmodified CSM and CFD Solvers · Multiplicative vertically (between levels), global coarse grid problem (MG-like) Additive horizontally: block-Jacobi / Schwarz](https://reader035.fdocuments.net/reader035/viewer/2022071011/5fc9513df88d897bf1617d9c/html5/thumbnails/21.jpg)
Conclusions
Introduction FEAST Co-processor integration Results Conclusions
![Page 22: GPU Acceleration of Unmodified CSM and CFD Solvers · Multiplicative vertically (between levels), global coarse grid problem (MG-like) Additive horizontally: block-Jacobi / Schwarz](https://reader035.fdocuments.net/reader035/viewer/2022071011/5fc9513df88d897bf1617d9c/html5/thumbnails/22.jpg)
Conclusions
Hardware-oriented numerics prevents existing codes being worthlessin a few years
Mixed precision schemes exploit the available bandwidth withoutsacrificing accuracy
GPUs as local preconditioners in a large-scale parallel FEM package
Not limited to GPUs, applicable to all kinds of hardware accelerators
Minimally invasive approach, no changes to application code
Excellent local acceleration, global acceleration limited by‘sequential’ part
Future work: Design solver schemes with higher accelerationpotential without sacrificing numerical efficiency
Introduction FEAST Co-processor integration Results Conclusions
![Page 23: GPU Acceleration of Unmodified CSM and CFD Solvers · Multiplicative vertically (between levels), global coarse grid problem (MG-like) Additive horizontally: block-Jacobi / Schwarz](https://reader035.fdocuments.net/reader035/viewer/2022071011/5fc9513df88d897bf1617d9c/html5/thumbnails/23.jpg)
Acknowledgements
Collaborative work with
FEAST group (TU Dortmund)
Robert Strzodka (Max Planck Institut Informatik)
Jamaludin Mohd-Yusof, Patrick McCormick (Los Alamos NationalLaboratory)
http://www.mathematik.tu-dortmund.de/~goeddeke
Supported by Deutsche Forschungsgemeinschaft, project TU 102/22-1, TU
102/22-2, TU 102/27-1, TU102/11-3
Introduction FEAST Co-processor integration Results Conclusions