Smoothed Particle Hydrodynamics @ BAW - SPH at BAW.pdf · Smoothed Particle Hydrodynamics @ BAW...

Smoothed Particle Hydrodynamics @ BAW

Eugenio RusticoReferat W5 – [email protected]

Workshop “Application of SPH in Environmental Engineering and Geosciences” – June, 30th - July, 1st – BAW Karlsruhe

SPH @ BAW - 2/58

“Application of SPH in Environmental Engineering and Geosciences” – June, 30th - July, 1st 2014

Outline

▶ GPUSPH

▶ Multi-GPU, multi-node

▶ Testcase: Fishpass

▶ Testcase: Shiplock

▶ Conclusions and future plans

SPH @ BAW - 3/58


Project: “Evaluierung und Adaptierung der SPH-Methode für wasserbauliche Fragestellungen an Bundeswasserstraßen”

“Evaluation and adaptation of the SPH method for hydraulic engineering problems on federal waterways”

Netzplan-Nr.: A39530570002

Projektleitung: Jankowski, Dr. J.

The project aims to increase the computational efficiency, development and validation of an existing SPH code (Smoothed Particle Hydrodynamics) for hydraulic engineering problems in which the traditional grid-based methods reveal its applicability limits.

GPUSPH

SPH @ BAW - 4/58


▶ Existing code: GPUSPH

▶ Tasks at BAW:

● Parallelization (multi-GPU multi-node)→

● Testcases (Fishpass, Shiplock)

▶ Testbed: new GPU nodes in brutus cluster

● 8 x BULLX B505 blades

● Each with 2 * Infiniband QDR network cards, latency 0.1µs

● 2 x K20m NVIDIA GPUs each, tot 16 devices

GPUSPH

SPH @ BAW - 5/58


ATHOS Consortium

GPUSPH is developed within the scope of the ATHOS research program, which brings together international partners interested in the development of “Advanced tools and methods for computational fluid-dynamics”.

The current ATHOS partners are:

● Istituto Nazionale di Geofisica e Vulcanologia (INGV), Sezione di Catania, Italy

● Johns Hopkins University (JHU), Baltimore (MD), USA

● Bundesanstalt für Wasserbau (BAW), Karlsruhe, Germany

● Univeristät für Bodenkultur Wien (BOKU), Wien, Austria

● Conservatoire National des Arts et Métiers, Paris, France (CNAM)

● EDF R&D, Chatou, France

GPUSPH

SPH @ BAW - 6/58


GPUSPH is an open source implementation of Smoothed Particle Hydrodynamics on GPUs (SPHERIC 3rd, 2008, Losanne)

http://www.gpusph.org

GPUSPH

http://www.gpusph.org/

SPH @ BAW - 7/58


GPUSPH Features include:

▶ Support for multiple SPH formulations, kernels, corrections, viscosities and boundary conditions

▶ Predictor-corrector scheme with adaptive timestep

▶ Surface particles detection, vorticity computation

▶ Floating objects, joints

▶ Support for planes, variable gravity and moving boundaries

▶ Periodic boundaries

▶ DEM topographies

▶ Homogeneous precision

▶ All features supported in multi-GPU and multi-node modes, with arbitrary domain decomposition

GPUSPH

SPH @ BAW - 8/58


GPUSPH Features do not include:

▶ ...

▶ Incompressibility

▶ Surface tension

▶ Solid erosion and breaking

▶ Easy and user friendly problem definition interface

▶ ...

GPUSPH

SPH @ BAW - 9/58


GPUSPH experimental branches

▶ Lava thermal loss and solidification

▶ Elastic solids [video]

▶ Pore water problems

▶ Sediment transport

GPUSPH

SPH @ BAW - 10/58


Why multi-device?

Multi-GPU

SPH @ BAW - 11/58


Why multi-device?

▶ High cost-effectiveness

▶ Performance improvement

▶ Bigger simulations

● Wide domains

● Higher resolution

Some applications require50+M parts

Multi-GPU

SPH @ BAW - 12/58


How multi-device?

▶ Spatial split

● Pros: easy, few transfers

● Cons: May unbalance

▶ List split

● Pros: easy, automatically balanced

● Cons: many sparse transfers

▶ Pipeline

● Pros: no extra memory

● Cons: many transfers, few computationally intense phases

Multi-GPU

SPH @ BAW - 13/58


Cells

▶ GPUSPH uses auxiliary grid to speed up neighbor list construction

▶ Cell side >= influence radius

▶ Indexing particles by means of a virtual grid speeds up the neighbor search: O(n²) O(c·n)→

Multi-GPU

SPH @ BAW - 14/58


Cells

▶ Idea: exploit the same grid for domain partitioning

▶ New in GPUSPH 3.0: store cell-relative coordinates instead of global ones to achieve homogeneous accuracy, regardless of domain size and resolution

Multi-GPU

SPH @ BAW - 15/58


Partitioning

Grid split

SPH @ BAW - 16/58


Partitioning

Grid split

SPH @ BAW - 17/58


Partitioning

Grid split

SPH @ BAW - 18/58


Partitioning

Grid split

Reading data across different devices is rather expensive and a naive approach requires to do it often.

Idea: each device keeps a read-only copy of the neighboring cells and updates it when needed.

→ memory overhead!

SPH @ BAW - 19/58


Partitioning

GPUSPH 2.0 (previous)

Previous policy: only split along slices, to have edge particles automatically compacted

SPH @ BAW - 20/58


Partitioning


Example split along X; requires compatible cell linearization function

SPH @ BAW - 21/58


Partitioning


This limitation is acceptable for multi-GPU but is too tight for multi-node (up to 256 devices)

SPH @ BAW - 22/58


Arbitrary decomposition

Example: river

Problem: unbalanced, wide sections, multiple sections per device

SPH @ BAW - 23/58



Example: river

Minimal number of neighbor devices, small cross sections

SPH @ BAW - 24/58



Arbitrary decomposition in GPUSPH 3.0

▶ The split function assigns each cell to a device

● Customizable by the user

● Several ready to use

▶ Special indices are assigned to particles before sorting

● This ensures compaction of the outgoing/ingoing cells

▶ Transfers are compacted in the smallest possible number of bursts and sent across the network

● Support for peer device copies, GPUDirect RDMA and asynchronous MPI transfers

SPH @ BAW - 25/58



Arbitrary decomposition examples

DamBreak3D, diagonal split

SPH @ BAW - 26/58



Arbitrary decomposition examples

OdeObjects, X split, 2 devices

SPH @ BAW - 27/58


Arbitrary decompositionArbitrary decomposition examples

Sample topography, Y split, 2 devices

SPH @ BAW - 28/58


Multi-node

Simulation modes

▶ Single-node, multi-GPU

● No MPI required

▶ Multi-node, single-GPU

● MPI, one or more processes per node, each controlling a GPU

▶ Multi-node, multi-GPU

● MPI, one multi-GPU process per node

SPH @ BAW - 29/58


Multi-node

Code excerpt

SPH @ BAW - 30/58


Multi-node

Usage examples

rustico@cluster$ ./GPUSPH device 0,1,2

rustico@cluster$ ./GPUSPH

rustico@cluster$ mpirun hostfile hosts.txt ./GPUSPH device 0,1

Running multi-GPU (assuming 3 CUDA devices are available):

Running single-GPU:

rustico@cluster$ ./GPUSPH device 1

Running multi-GPU multinode (2 CUDA devices per node):

SPH @ BAW - 31/58


Performance

Performance

1.9x speedup on 2 devices 5.1x speedup on 6 devices12.6x speedup on 16 devices

SPH @ BAW - 32/58


Performance

Performance

Reaching the ideal speedup is not possible in reality

SPH @ BAW - 33/58


1.9x speedup on 2 devices 5.1x speedup on 6 devices12.6x speedup on 16 devices

Performance

Performance

Amdahl's lawlimit (5.5x)

SPH @ BAW - 35/58


“A fishpass (…) is a structure on or around artificial and natural barriers (such as dams, locks and waterfalls) to facilitate diadromous fishes' natural migration.” (Wikipedia)

Fishpass

SPH @ BAW - 36/58


Geometry of walls is designed to produce turbulent mixing, reducing kinetic energy of main flow.

Fishpass

SPH @ BAW - 37/58


Slower main stream makes it easier for fishes to swim upstream, recirculation areas allow fishes to rest

Fishpass

SPH @ BAW - 38/58


CFD for fishpass design

▶ Evaluate stream velocity and recirculation areas

● Fishes are unable to swim upstream if current too strong

● Recirculation areas are needed to allow fishes to rest

▶ Design appropriate wall geometry and aperture sizes

● Constraint: aperture large enough to let fishes pass through comfortably

● Small geometry changes can have large impact on recirculation and velocity of main stream

Fishpass

SPH @ BAW - 39/58


Fishpass scaled model @ BAW

▶ 9 pools, 1m long and 0.785m wide

▶ Inflow rate: 20l/s

▶ Wooden walls, plexiglas floor

▶ ~16m total length

▶ used to model different geometries for fish-passes on the Neckar river (Germany)

▶ 1:4 scale

Fishpass

SPH @ BAW - 40/58


Fishpass model with SPH

▶ Fully parametrized

▶ Infinite planes for sides, Lennard-Jones particles for walls

▶ Pseudo-inflow with velocity field

▶ Wendland kernel, artificial viscosity, MLS velocity correction

Fishpass

SPH @ BAW - 41/58



Preliminary test: periodic. Result: too fast

▶ Good stream shape and recirculation!

Fishpass

SPH @ BAW - 42/58



Preliminary test 2: periodic with velocity fields. Result: unstable

Fishpass

SPH @ BAW - 43/58



Preliminary test 3: inflow + ramp. Result: good stream, but “nuclear reactions” happen in high resolution

Fishpass

SPH @ BAW - 44/58



Pseudo-inlet for particle generation

Fishpass

SPH @ BAW - 45/58



GPUSPH 3.0 introduced:

▶ Homogeneous accuracy (no position quantization) [Hérault et al., SPHERIC 2014]

▶ Compact neighbor list (more particles fit on a GPU)

▶ Multi-node (100+ millions particles possible on 16 GPUs cluster)

Numerically accurate, high resolution fishpass is now possible

Fishpass

SPH @ BAW - 46/58



Fishpass

SPH @ BAW - 47/58


Comparison with laboratory measures – physical model

Fishpass

SPH @ BAW - 48/58


Comparison with laboratory measures – SPH model

Fishpass

SPH @ BAW - 49/58


Comparison with laboratory measures – water levels

Fishpass

SPH @ BAW - 50/58


Kinematic viscosity shows lower and more noisy water depths. Now testing SPS

Fishpass

Pool 6, gage A

Pool 6, gage B

ARTIFICIAL KINEMATIC

SPH @ BAW - 51/58


Fishpass: ongoing & future

▶ Very good stream shape and direction

▶ Artificial viscosity and Lennard-Jones boundaries introduce too many artifacts

▶ Switching to semi-analytical boundaries

Fishpass

SPH @ BAW - 52/58


Shiplock

Shiplock

▶ The large ship locks of the Kiel-Canal is undergoing a significant renovation and will use a through-the-gate filling system

▶ CFD modeling performed with VOF at BAW: Evaluation of ship forces for a through-the-gate filling system, Thorenz, C.; Anke, J., Smart Rivers 2013

SPH @ BAW - 53/58


Shiplock

Shiplock

▶ CFD modeling to evaluate

● Filling and emptying times (efficiency)

● Forces acting on the ships (safety)

▶ Modeling a floating ship is non-trivial

SPH @ BAW - 54/58


Shiplock

Shiplock

▶ High number of particles required

● High domain/detail ratio: smallest geometry is 1.3m long

● 338m x 45m size, 12.4~16m water depth 4·10→ 5 m³ water

▶ Floating ship (preliminary model: cylinder)

▶ ~1,000s simulated time

SPH @ BAW - 55/58


Shiplock

Shiplock

▶ ~30M particles on 8 nodes, ARTVISC, inlet with time-dependent rate

SPH @ BAW - 56/58


Shiplock

Shiplock

▶ Still undergoing evalution

SPH @ BAW - 57/58


GPUSPH experience at BAW

▶ Currently investigating first, promising results with fishpass and shiplock testcases

▶ Semi-analytical boundaries: ready, switch in progress for the testcases

▶ S.A. pressure inlet: in development

● Affects models! (es. tank vs. inlet in Shiplock)

▶ Load balancing: in development (NP-complete with arbitrary decomposition)

▶ Improve user interface

▶ Many experimental branches out there

Conclusions

SPH @ BAW - 58/58


...that's all, folks!

Conclusions

Thank you

Smoothed Particle Hydrodynamics @ BAW - SPH at BAW.pdf · Smoothed Particle Hydrodynamics @ BAW...

Documents

Transcript of Smoothed Particle Hydrodynamics @ BAW - SPH at BAW.pdf · Smoothed Particle Hydrodynamics @ BAW...