Lattice Boltzmann simulations on heterogeneous CPU-GPU...

Computer Science X - System Simulation Group Harald Köstler (harald.koestler@fau.de)

Lattice Boltzmann simulations on heterogeneous CPU-GPU

clusters H. Köstler, Ch. Feichtinger

2nd International Symposium

“Computer Simulations on GPU” Freudenstadt 29.05.2013

Contents

Motivation

waLBerla software concepts

LBM simulations on Tsubame

Future Work

Computational Science and Engineering @ LSS

Applications • Multiphysics • fluid, structure • medical imaging • laser

Applied Math • LBM • multigrid • FEM • numerics

Computer Science • HPC / hardware • Performance

engineering • software

engineering USE_SweepSection( getLBMsweepUID() ) USE_Sweep() swUseFunction(„LBM",sweep::LBMsweep,FSUIDSet::all(),hsCPU,BSUIDSet::all()); USE_After() //Communication

Problems

Hardware: Modern HPC clusters are massively parallel Intra-core, intra-node, and inter-node

Software: Applications become more complex with increasing computational power

More complex (physical) models

Code development in interdisciplinary teams

Algorithm: Many variants exist Components and parameters depend on computational domain or grid, type of problem, …

WALBERLA Applications

waLBerla: parallel block-structured grid framework

waLBerla @ GPU

Geometric multigrid solver on Tsubame

Computational Steering (VIPER)

CFD, fluid-structure interaction

1000 1500 2000 2500 3000 3500

unknowns in million

Boltzmann equation

Mesoscopic approach to solving the Navier-Stokes equations

Boltzmann equation describes the statistical distribution of one particle in a fluid

f is the probability distribution function (PDF), the particle velocity, and Ω(f) is the change due to collision

Models behavior of fluids in statistical physics

Lattice Boltzmann Method (LBM) solves the discrete Boltzmann equation

)(f f ft Ω=∇⋅+∂ ζ

Particulate Flow Simulation

D3Q19 LBM cell Collide and Stream

amF ⋅=α⋅= JM

WALBERLA CPU-GPU cluster software concepts

waLBerla framework

Main goal: provide a massive parallel and efficient software framework for multi-physics simulations

WaLBerla is mainly designed for HPC clusters

waLBerla (C++) Code management,

standard implementations

Low-level kernels for optimized architecture-

specific computations (in

C++, CUDA, Assembler)

waLBerla: Block concept

waLBerla: Sweep concept

Challenges on heterogeneous clusters I

Problem: Description of the heterogeneous compute resources Solution: Description of all compute components per compute node in the input file

Problem: Management of the communication and compute kernels for each architecture Solution: Kernel management based on meta data

Challenges on heterogeneous clusters II

Problem: Common communication interface Solution: Data exchange via communication buffers also for intra node communication

Problem: Minimization of the heterogeneous communication overhead Solution: Overlapping of work and communication, non-uniform domain decomposition, and intra-node communication in z-dimension

waLBerla: Communication concept

Overlapping of work and communication

WaLBerla: Subblocks

Assumption: A block corresponds to a (shared-memory) compute node

Can possibly be heterogeneous (CPU + GPU)

Distributed memory communication (via MPI) is not required within one block

Divide one block into subblocks of different sizes for (static) load balancing

Subblocks map to (local) devices

Domain decomposition on one compute node

RESULTS LBM Simulations on Tsubame 2.0

Tsubame 2.0 in Japan

Compute nodes: 1442

Processor: Intel Xeon X5670

GPU: 3 x Nvidia Tesla M205

Peak performance:

2.2 PFlop/s

633 TB/s memory bandwidth

LINPACK performance: 1.2 Petaflops

Power consumption: 1.4 MW

Interconnect: QDR Infiniband

Performance Engineering

Create performance model

Identify performance bottlenecks

Create problem-specific, hardware-

dependent, and highly efficient kernel

Integrate them in software framework

Algorithm Hardware

Input Algorithm: LBM kernel

Generic Implementation

Hardware information (bandwidth, peak performance)

Assumption

Computation time limited by memory bandwidth and instruction throughput

Communication time limited by network bandwidth and latency (for direct and collective communication)

Performance Model I

),max( ,,,, MPIcommGPUCPUcommbufferinnercompoutercomptotal tttttt +++=

Single node performance on Tsubame

Machine balance

Code balance

Lightspeed estimate (if l < 1 code is bandwidth limited)

Performance Model II

eperformancpeak bandwidth esustainabl

BBl ,1min

200304

FLOPS executed no.stored and loaded bytes no.

Single Compute Node Performance I

Single Compute Node Performance II

Single Compute Node Performance III

Single Compute Node Performance IV

Communication model

),max( ,,,, MPIcommGPUCPUcommbufferinnercompoutercomptotal tttttt +++=

Communication time for one message depends on size of message s

number of messages x that are concurrently transferred over the communication link (communication pattern)

type of communication link ω

relative position of the communication partners e.g.intra- or inter-node communication p

Weak scaling, 3 GPUs per node

Strong scaling, 3 GPUs per node

Test case: Packed bed of hollow cylinders

Porous media: 100x100x1536, 1D dom. decomp.

Porous media: 100x100x1536, 1D/2D/3D

Porous media: 256x256x3600, 1D/2D

Current Work

Focus in waLBerla currently on Juqueen and SuperMUC

Future Work

Tests on Nvidia Kepler cluster

Programming paradigms on future HPC clusters?

Code generation techniques to improve portability

Dynamic load balancing

Lattice Boltzmann simulations on heterogeneous CPU-GPU...

Documents

Transcript of Lattice Boltzmann simulations on heterogeneous CPU-GPU...

Genießerwege rund ums Walberla 2009 - RunningSocks · Geniesserwege rund ums Walberla_2012.qxd:Genießerwege rund ums Walberla 2009 05.08.2012 22:55 Uhr Seite 3 - Sollte Ihnen der

SYLVIA HUBELE, HOBBYSPORTLERIN Auf dem Walberla wird ...vom+Sofa6.pdfPfad zum Walberla hinauf trans-portierten und anschließend akri-bisch mit „hau-ruck“ und einigen Maß Bier

New LiterNatur am Walberla · 2020. 5. 5. · LiterNatur am Walberla Der Frühling am Walberla hat seinen ganz besonderen Zauber! Tauche ein mit uns, in das Erwachen von Mutter Erde.

ENABLING HPC APPLICATIONS FOR SVE€¦ · WALBERLA SIMULATOR [WaLBerla] Flexible LBM simulation application Support of a broad range of LBM models Optimised for different architectures

sv PRETZFELO 1946 SG Walberla HEIM Spieltag… · sv PRETZFELO 1946 1962 . Title: SG Walberla Spieltag.cdr Author: rodamei Created Date: 10/5/2015 12:37:45 PM

Kirschenblüte „Rund ums Walberla“ Sonntags-Schlemmens/angeb67.53.9.B.pdf · Mittwoch, 22. April 2020 Abf. 9.00 – Rückk. 19.00 Uhr Die Kirschenblüte „Rund ums Walberla“

Wanderung Rund um das Walberla - Sockenqualmer · Walberla genannt. Von dem Zeugenberg hat man einen unvergleichlichen Blick in das untere Wiesenttal und die Regnitzebene. Der mächtige

C M Y CM MY CY CMY K - Walberla · 2007. 9. 2. · Schlaifhausen, Leutenbach laden ein zum Mitfeiern 20 Jahre Naturschutzgebiet Walberla/Ehrenbürg. Freitag, 21. Sept. 2007 19.00

Ludwig Boltzmann BZ-1homepages.gac.edu/~anienow/CHE-371/Lectures/Boltzmann...Ludwig Boltzmann BZ-1 Boltzmann is well-known as a chemical theorist – he worked on topics ranging from

Mach Boltzmann

Deterministic Methods for the Boltzmann Equation€¦ · The Boltzmann equation (4) with nonzero Lorentz force Lis sometimes called the Maxwell-Boltzmann system (or the Poisson-Boltzmann

Lattice Boltzmann for CFD and beyond - FAU · waLBerla: Widely applicable LB solver from Erlangen (Uli Rüde’s group) uses “prototype” kernel “Patch-based” approach Large

Ludwig Boltzmann BZ-1 - Gustavus Adolphus Collegehomepages.gac.edu/~anienow/CHE-371a/Lectures/Boltzmann...Ludwig Boltzmann BZ-1 Boltzmann is well-known as a chemical theorist – he

WaLBerla::Core: An Overview

Partnership Initiative Computational Science –Using HPC ... · waLBerla IBM 128000Lattice Boltzmann 5.6 90 LAMMPS IBM 128000Molecular Modelling 5.6 90 APES IBM 64000CFD 6 47 BQCD

Thermodynamics Maxwell-Boltzmann Distribution Second Lawnebula2.deanza.edu/~lanasheridan/4C/Phys4C-Lecture21.pdfReminder: Maxwell-Boltzmann speed distribution. The Boltzmann distribution

Experimental Verification of theExperimental Verification ... · waLBerla Widely applicable ll attice Boltzmann from Erlangen Homogeneous Mixture FlowsHomogeneous Mixture Flows Simulation

Energy Efficient data centers A holistic approach and best ... · waLBerla Lattice Boltzmann Musubi Lattice Boltzmann CIAO CFD, Combustion Vertex3D Stellar Astrophysics LS1-Mardyn

Chanwoo - Boltzmann

Boltzmann Concept