04.04.2017:: ::

::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: :::::

::

Research towards Exascale at HLRS

Joseph Schuchart – Scalable Programming Models and Tools, HLRS

04.04.2017:: ::

::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: :::::

::

OVERVIEW - HLRS

Research at HLRS 2

04.04.2017:: ::

::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: :::::

::

High Performance Computing Center Stuttgart – HLRS

Research at HLRS 3

04.04.2017:: ::

::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: :::::

::

High Performance Computing Center Stuttgart – HLRS

Research at HLRS 4

04.04.2017:: ::

::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: :::::

::

Bundeshöchstleistungsrechenzentrum Stuttgart – HLRS

• 1st German national Supercomputing Center, Founded 1996

• Central unit of the University of Stuttgart

• Open to Eurepean users since 2004 through PRACE (Partnership for Advanced

Computing in Europe)

• Funding per year:

– 3,9 M€ State funding

– 5,0 M€ Project funding

• Staff:

– ~ 25 permanent Staff

– ~ 60 Project Staff

– ~ 20 Research Assistants

• > 7500 TFlops installed computing power in three main systems

• Comprehensive user training

Research at HLRS 5

04.04.2017:: ::

::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: :::::

::

Education & Training• Education

• Training (20 courses – 600 participants)

• 4-5 International Tutorials at conferences

• 4-5 own conferences and workshops at HLRS

• PRACE Advanced Training Center together with JSC and LRZ

• PRACE Industrial Training Center

• New Training Center (2017)

Research at HLRS 6

04.04.2017:: ::

::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: :::::

::

Research Departments• Applications and Visualisation

– Visualisation

– Scalable Programming Models & Tools

– Scalable Computing & Optimisation Methods

– Service Management & Business Processes

• Software & Systems

– Numerical Methods & Libraries

– Project & User Management, Accounting

• Staff Unit

– Philosophy of Science & Technology of Computer Simulation

Research at HLRS 7

09.04.2017:: ::

::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: :::::

::

SYSTEMS

Research at HLRS 8

09.04.2017:: ::

::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: :::::

::

HLRS - Systems

2015 Cray XC40 “Hazel Hen”• 41 Cabinets• 7712 Compute nodes

– 2 sockets– 12 Cores per socket– 128 GB Memory

• 185,088 Cores, Intel Haswell• 7.420 PF/s Peak performance• ~ 5.640 PF/s Sustained performance• ~ 10 PB Disk Space• Power consumption: ~ 3.2 MW

• Cray Aries Network • Cray-Linux-Environment • Cluster Compatibility Mode (For ISV Applications)

Research at HLRS 9

09.04.2017:: ::

::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: :::::

::

HLRS - Systems

NEC PC-Cluster• Mainly industrial usage with ISV codes• Scientific Linux 6.6• Redundant System in two computing rooms• Nobelstraße 19

– 650 Nodes with various architectures• Sandy Bridge, Haswell• AMD Interlagos

– 12 GB – 1TB Main memory per node– ~ 500 TB Workspace Filesystem– Infiniband Network

• Almandring 30– 192 Nodes with Intel Sandy Bridge Architecture– ~ 128 TB Workspace Filesystem– Infiniband Network

Research at HLRS

Nobelstraße19

Almandring 30

10

09.04.2017:: ::

::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: :::::

::

HLRS - Systeme

NEC SX-ACE

• 64 Nodes

– 4 Cores per Node

– 64 GB Main memory per Node

– Compact and power efficient construction

• 16.384 TFLOPS Peak Performance

• 250 TB NEC Scalable Technology File System (ScaTeFS)

HPSS Datamanagement System

• Hierarchical Storage Management System (HSM)

• >12 PB Storage capacity on tape (max. 30PB)

• 10,000 Tapes (handled by tape robot)

• ~500 TB HDD Cache

Research at HLRS

SX-ACE Node Card

NEC SX-ACE Single Rack

HPSS

11

09.04.2017:: ::

::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: :::::

::

RESEARCH AREAS

Research at HLRS 12

09.04.2017:: ::

::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: :::::

::

HLRS RESEARCH

Research at HLRS 14

09.04.2017:: ::

::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: :::::

::

Mont Blanc: Model-centric debugging with

• Traditional debuggers aim to cover a wide range of programming languages and models and thus present information using the lowest common denominator, i.e. in terms of low-level concepts

• Model-centric debugging allows using the same language and semantics for debugging and programming.

• Temanejo is a graphical debugger for task-based programming models as e.g. OpenMP or OmpSs

Temanejo supports direct interaction with tasks (step into/over, pause, continue) and dependencies (adding, removing, inspecting).

Research at HLRS 15

09.04.2017:: ::

::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: :::::

::

CoE POP: Performance Optimization and Productivity

Goal: understanding common performance issues of real codes

Approach: analyze as many codes as possible

Incentive for users: offer performance analysis and optimization services

Service levels:

• Performance audit and analysis

• Proof-of-concept implementation of optimizations

• Training on performance tools Ateles CFD code50% performance improvementhttps://pop-coe.eu/blog/pop-proof-of-concept-study-leads-to-nearly-50-higher-performance

Research at HLRS 16

09.04.2017:: ::

::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: :::::

::

Simulation software & Application Areas

• Topics

– Energy

– Climate Change & Environment

– Health & The Aging Society

– Mobility in the 21st Century

Research at HLRS 18

09.04.2017:: ::

::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: :::::

::

CONVECTION PERMITTING SEASONAL LATITUDE-BELT SIMULATION

Applications in High Performance Computing – Climate Simulations

Research at HLRS 19

09.04.2017:: ::

::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: :::::

::

Convection permitting seasonal latitude-belt simulation

• Application by:– Intitute of Physics and Meteorology – University of

HohenheimT. Schwitalla, K. Warrach-Sagi, and V. Wulfmeyer

• Purpose– Improve long range forecasts with Limited Area Models

(LAM)– Reduce model assumptions– Overcome Resolution Problems

• Model setup– Full Belt between

20° North and 65° North– Resolution of 0.03° (3.3km)– Simulation period

July and August 2013

Research at HLRS 20

09.04.2017:: ::

::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: :::::

::

Convection permitting seasonal latitude-belt simulation

• Simulation with community code WRF

• 84,000 compute cores

• Hybrid OpenMP / MPI execution

• 84 h (3,5 days) Runtime

• 450 TB Result Data

• I/O

– 3D data every 30min

– Diags every 15min

• 36 h (1,5 days) I/O Runtime

Research at HLRS 21

09.04.2017:: ::

::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: :::::

::

Results

Research at HLRS

Low pressure system development over the North Atlantic

22

09.04.2017:: ::

::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: :::::

::

Results - Typhoon Soulik

Research at HLRS

• Emerged July 2013

• Made landfall late on July 12th in northern Taiwan

• Time of emergence and landfallwas calculated correctly

• Landfall was calculatedwith 100 km error

• Feature is not visible / reproduciblewith global climate models at 50 kmor 100 km resolution.

23

09.04.2017:: ::

::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: :::::

::

AXIAL FAN TURBULANCESIMULATION

Applications in High Performance Computing – Mechniacs Simulations

Research at HLRS 24

09.04.2017:: ::

::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: :::::

::

Prediction of the Turbulent Flow Field Around a Ducted Axial Fan

• Cooperation with the Institute for Aerodynamicsat the RWTH Aachen University (AIA)

• Funded through industrial research partnership

Goals:

• Development of a reliable prediction tool for the noisegeneration of ducted axial fans

• Development of reliable turbulence models forindustrial applications

Research at HLRS

1) Copyright: Institute of Aerodynamics, RWTH Aachen University2) „Luefter“ Das Original wurde von Huha in der Wikipedia auf Deutsch Lizenziert unter CC BY-SA 3.0 über Wikimedia Commons - https://commons.wikimedia.org/wiki/File:Luefter.jpg#/media/File:Luefter.jpg

1)1)

2)

25

09.04.2017:: ::

::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: :::::

::

Prediction of the Turbulent Flow Field Around a Ducted Axial Fan

• Noise created through

– Interaction of fan and turbulances

– Streaming phenomenae at Tip Gap

Probleme:

• Complex geometries

• Turbulent field with

– Unfavorable pressure gradients

– Heavily bound streamlines

• Results of Computational Aeroacoustics (CAA) depend heavily on quality of CFD results

Research at HLRS

1) Copyright: Institute of Aerodynamics, RWTH Aachen University2) „Luefter“ Das Original wurde von Huha in der Wikipedia auf Deutsch Lizenziert unter CC BY-SA 3.0 über Wikimedia Commons - https://commons.wikimedia.org/wiki/File:Luefter.jpg#/media/File:Luefter.jpg

2)

Tip Gap

Berechnungsgitter imBlatt-Mantel Spalt

Berechnungsgeometrie

1)

1)

26

09.04.2017:: ::

::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: :::::

::

Prediction of the Turbulent Flow Field Around a Ducted Axial Fan

• Results • RANS: Reynolds Averaged NavierStokes Method

– Commercial ISV codes

– „Workstation result“

• LES: Large Eddy Simulation

– Simulated using ZFS

– 92,000 compute cores

– 110 walltime

– 80 TB result data

– 1 Billion grid cells

– 320,000 time steps for 4 rotations

Research at HLRS

All figures: Copyright: Institute of Aerodynamics, RWTH Aachen University

27

09.04.2017:: ::

::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: :::::

::

Challenges on the path to Exascale (at HLRS)

Research at HLRS

Exascale

UsablePerformance

Infra-structure

ApplicationScalability

Productivity

I/O

Training

28

09.04.2017:: ::

::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: :::::

::

Thank you for your attention!

High Performance Computing Center Stuttgart