Wolfram Schmidt et al- FEARLESS: A new modelling approach for turbulent astrophysical flows

8/3/2019 Wolfram Schmidt et al- FEARLESS: A new modelling approach for turbulent astrophysical flows

1/36

FEARLESSFEARLESS

A new modelling approachA new modelling approach forfor

turbulentturbulent astrophysicalastrophysical flowsflows

Orion Nebula (VLA)


2/36

Wolfram SchmidtWolfram Schmidt

J. C.J. C. NiemeyerNiemeyer, L., L. IapichinoIapichino

A. Maier, M.A. Maier, M. HuppHupp, Ch., Ch. FederrathFederrath, J., J.AdamekAdamek

InIn collaborationcollaboration withwith

Ch. Klingenberg (Inst. f. Mathematik, Univ. WCh. Klingenberg (Inst. f. Mathematik, Univ. Wrzburg)rzburg)

A.A. KritsukKritsuk(Laboratory f.(Laboratory f. CompComp..AstrophysicsAstrophysics,, UCSD)UCSD)


3/36

DEISA TrainingBarcelona, Spain

Wolfram SchmidtUniversitt Wrzburg, Germany

3

AstrophysicalAstrophysicalTurbulenceTurbulence

Planetary and stellar convective boundary layers

Heat transport by convection inside stars Turbulent thermonuclear combustion in

supernovae

Gas flow in accretion disks Turbulence in the interstellar medium Hot gas in galaxy clusters


4/36



4

TurbulenceTurbulence FundamentalsFundamentals

Fluid motion becomes increasingly chaotic

through non-linear energy transferVortices (eddies) develop on a multitude of scales

Turbulent dynamics is vortex stretching in threedimensions


5/36



5

TurbulenceTurbulence FundamentalsFundamentals

Fluid motion becomes increasingly chaotic

through non-linear energy transferVortices (eddies) develop on a multitude of scales

Turbulent dynamics is vortex stretching in threedimensions


6/36



6

So, natralists observe, a fleaHath smaller fleas that on him prey;

And these have smaller yet to bite em

And so proceedad infinitum

Thus every poet[fluid dynamicist], in his kind,

Is bit by him that comes behind.

TheTheTurbulenceTurbulence CascadeCascade

Lewis Fry Richardson


7/36



7

Large Eddy SimulationLarge Eddy Simulation

# degrees of freedom: E(k)

K kL

k-5/3

Reynolds number:

turb.: Re ~ 1000, terrestr.: Re ~ 107, astrophys.: Re ~ 1014

Supercomputers manageN~ 109, i.e. Re ~ 104


8/36



8

Large Eddy SimulationLarge Eddy Simulation

Static numerical grid of given resolution

=10-3

L 0.01LComputation of flow dynamics at scales < l < L

Subgrid scale model accounts for turbulent eddiesof size l <

SGS turbulence stress ~(SGS turbulent viscosity) * (rate of strain)


9/36



9

SupersonicSupersonicTurbulenceTurbulence

Mach number:

Compressibilitybecomes important

ifM a ~ 1

For M a > 1 , shockfronts dominateover large eddies


10/36



10

AdaptiveAdaptive MeshMesh RefinementRefinement

Hierarchy of dynamically created grids of

varying resolution n= L/Nn= L/(n

N0),where = 2 or 4

Grid geometry adapts to flow structureKritsuke

tal.(2006

)


11/36



11

ButButTurbulenceTurbulence isis

SpaceSpace--FillingFilling, Right?, Right?Homogeneous turbulence is space-filling from

the view point of the ensemble average(Kolmogorov theory, E(k) ~ k -5/3)

However, turbulence is intermittent

At any instant of time, dissipative structures(turbulent eddies, shocks) are concentrated in

regions offractal dimension D less than 3

Challenge: Keep track of turbulent flow structure via

appropriate refinement criteria in AMR simulations


12/36



12

AMR + SGSAMR + SGS

= FEARLESS= FEARLESS

effectively resolved subgrid

L 2 K

Resolve shocks and collapsing regions with AMR

SGS model treats asymptotically isotropic turbulence

3 41

Fluidluid mmEchanicschanics withwithAdaptivelydaptively

Refinedefined Largearge Eddyddy SimulationimulationS


13/36



13

DEISA Project GoalDEISA Project Goal

In our DEISA project, we investigated the

applicability of AMR to forced supersonicturbulence by implementing new refinementcriteria into the Enzo code and comparing it tostatic grid simulations.

Enzo is an Open Source C++/Fortran AMRcode developed by the Laboratory forComputational Astrophysics at The University

of California in San Diego.


14/36



14

Big DEISA RunsBig DEISA Runs

Box with periodic boundary conditions

Stochastic forcing drives turbulent flow Characteristic length scale ~ L= box size

Intitial condition: const. density, gas at rest

Force field stirs and compresses gas over time scale T

RMS velocity asymptotically approaches ~ V = L/T

Static grid,N = 7683

= 452984832 cubic cells ~ 10000 time steps for sufficient relaxation

~ 100000 CPU-h required for one simulation!


15/36



15

DEISADEISA ResourcesResources

DEISA Extreme Computing Initiative (DECI): Free CPU-timeand porting/optimization support for challenging projects

FEARLESS was assigned to the Dutch computing centerSARAas an execution site, with the German LeibnizComputing Center (LRZ) managing correspondence and

support

Jobs were submitted to SARA using the UNICORE client

SARA-ASTERmachine features:

SGI Altix 3700 with 512 CPUs in four partitions

Maximum #CPUs = 126 for MPI programs

Linux OS


16/36



16

TechnicalTechnical Problems andProblems and

SolutionsSolutionsLessonsLessons LearnedLearned Initial difficulties with Grid certificates, UNICORE

and the SARA runtime environment Fixed in collaboration with SARA and LRZ

Make sure to set aside some time for setting up

and getting used to the grid middleware thisis not simply a matter of logging into a remotesystem via ssh.


17/36



17


SolutionsSolutionsLessonsLessons LearnedLearnedEnzo did not respond well to some of the new

features implemented (segfaults, libraryincompatibilities) and to the new hardware

A number of debug iterations on a LRZ Altix machine

in cooperation with the LRZ DEISA team helped to fixthe problems

Remote debugging is very cumbersome if unsure

whether your code works on the target platform,ask your computing center for a similar (local)machine to do the testing on.

T h i l bl d


18/36



18


SolutionsSolutionsLessonsLessons LearnedLearnedFEARLESS uses a full quarter of the target

machine, despite decreasing the computationaldomain from 10243 to 7683

Special arrangement with SARA allows high

priority runs for long uninterrupted periodsFlexibility on both sides is in order, as long as

the virtualization of the computing resources is

not realized in its entirety.

T h lT h i l bl dP bl d


19/36



19


SolutionsSolutionsLessonsLessons LearnedLearnedTotal amount of primary output data: ~1 TB.

After post processing and visualisation, ~4 TBhave to be transferred to LRZ and archived

GridFTP used (UNICORE and scp too slow, GPFS

not yet available for the machines involved) takes about 20 hrs of uninterrupted transfer atthe 55 MB/s network speed reached

You may not have your results available locallyimmediately if major amounts of data areproduced, think about where you need them andmaybe ask your computing center for a net speed

test.

T b lT b l EE


20/36



20

TurbulenceTurbulence EnergyEnergy

andandVorticityVorticity Isothermal EOS

Mach numberV/c0 = 2.89

T b lT b l EE


21/36



21




T b lT b l EE


22/36



22




T b lT b l EE


23/36



23




T b lT b l EE


24/36



24




T b lT rb l EE r


25/36



25




T b lT rb lence EEnergy


26/36



26




T rb lenceTurbulence EnergEnergy


27/36



27





28/36



28

MassMass DensityDensity

Mostly compressive

forcing generatescompact dense regions

TheThe QuestQuest forfor TurbulenceTurbulence


29/36



29

TheThe QuestQuest forforTurbulenceTurbulence

RefinementRefinement CriteriaCriteriaSmall test runs:N0 = 96

3 root grid, 1 refined level

+N = 192

3

static grid simulation for comparison Temporal evolution of global statistical moments

Computation ofprobability density functions

3D visualisationsProduction runs:N0 = 192

3 root grid, 1-2 refinedlevels

Single data dump comprises up to 20000 outputfiles (proportional to number of refined regions)

Postprocessing and analysis quite laborious

TheThe QuestQuest forfor TurbulenceTurbulence


30/36



30

TheThe QuestQuest forforTurbulenceTurbulence

RefinementRefinement CriteriaCriteriaThe conventional approach is to refine grids in

the vicinity ofsteep gradientsA priori specification of approriate thresholds for

gradients is difficult

We investigated an alternative approach: Monitoring ofvorticity and rate of compression

Thresholds based on the regional variability

Only peaks of turbulence production and gascompression should trigger refinement!

GettingGetting Hold ofHold of SupersonicSupersonic


31/36



31

GettingGettingHold ofHold ofSupersonicSupersonic

TurbuleneTurbulenewithwithAMRAMRvorticity squared mass density

GettingGetting Hold ofHold of SupersonicSupersonic


32/36



32

GettingGettingHold ofHold ofSupersonicSupersonic

TurbuleneTurbulenewithwithAMRAMR probabilitydensityfunction

ofm

assdensity

AMRAMR worksworks

AstrophysicalAstrophysical ApplicationApplication::


33/36



33

AstrophysicalAstrophysicalApplicationApplication::

GalaxyGalaxyClusterCluster SimulationsSimulationsSmall clusters (subclusters) of galaxies

occasionally fall into pontential wells of largeclusters

Infall produces bow shockand vortex tail in

surrounding intercluster medium (hot dilute gas)Wide range of different scales necessitates AMR

Application of different refinement criteria inthree-dimensional simulations

AstrophysicalAstrophysical ApplicationApplication::


34/36



34

AstrophysicalAstrophysicalApplicationApplication::

GalaxyGalaxyClusterCluster SimulationsSimulationsIapichino & Adamek (2007)

conventionalconventional FEARLESSFEARLESS


35/36



35

ResumeResume

I. Since DECI policy does not allow projects to beextended, and computation resources are fixed, it has

proven difficult to complete the development andimplementation of novel techniques within the project.II. Ideally, the code used and the problem to be solved

computationally should be well understoodbeforehand, thereby enabling accurate resourceestimates in the project proposal.

III. Practically, more flexibility in the usage of the DEISA

resources for tackling problems at the frontiers ofcontemporary research would be desirable, i.e.completely virtualized resources. While this is the verypoint of distributed access to supercomputers, it seems

to be quite far off.

A k l d


36/36



36

AcknowledgementAcknowledgement

We thank the DECI teams

from SARA and LRZ for

their extensive support!

Wolfram Schmidt et al- FEARLESS: A new modelling approach for turbulent astrophysical flows

Documents

Transcript of Wolfram Schmidt et al- FEARLESS: A new modelling approach for turbulent astrophysical flows