Production Version of Flash : 1.61

An Accelerated Strategic Computing Initiative (ASCI) Academic Strategic Alliances Program (ASAP) Center

at The University of Chicago

The Center for Astrophysical Thermonuclear Flashes

Production Version of Flash: 1.61Paul Ricker

University of Chicago

The Flash Code GroupA. Caceres, A. Calder, T. Dupont, J. Dursi, B. Fryxell, T. Linde, A. Mignone,

K. Olson, P. Ricker, R. Rosner, K. Riley, A. Siegel, F. Timmes, H. Tufo,N. Vladimirova, G. Weirs, K. Young, M. Zingale

ASCI Flash Center University of Chicago

Overview

Flash 1.61 in context

Algorithmic architecture

Automated verification

Summary


Flash 1.61 in context


Flash framework evolution

FLASH 0.0Initial

FLASH 1.0Organization

FLASH 1.6Architecture,Optimization

FLASH 2.0Data


Physics modules (* - new this year) Compressible hydrodynamics

Piecewise-parabolic method (PPM) Weighted essentially non-oscillatory (wENO) * Tadmor central-difference scheme *

Compressible magnetohydrodynamics * Total variation diminishing (TVD) *

Equations of state (EOS) Partially degenerate stellar EOS (Coulomb corrections) * Mixture of perfect gases *

Source terms Nuclear burning - variety of reaction networks * Single-mode turbulent energy injection *

Flash capabilities


Flash capabilities

Physics modules (continued) Gravitational field

Externally imposed (constant, plane parallel *, point source *) Self-gravity *

Multipole summation * Multigrid * Couchman FFT method *

Diffusion * Thermal diffusion (explicit) * Radiation (implicit) *

I/O formats Hierarchical data format (HDF) version 4 and 5 *


Implementation 83,000 lines of FORTRAN 90 + C

~ 24,000 in driver + mesh package ~ 59,000 in physics modules

Block-structured adaptive mesh using PARAMESH Parallelization using MPI I/O using HDF4/5

Ports ASCI platforms (Red, Blue, Nirvana) Cray T3E Linux clusters (Intel & Alpha)

Flash 1.61 implementation


Algorithmic architecture


Framework design requirements Performance

Must be able to do the largest problem possible Modularity

Must be easy to switch initial/boundary conditions Must be easy to add/change solvers

Ease of adoption Should work with existing F77 routines Should not require many extra packages Must be portable Should use standard data formats

Ease of experimentation Must be easy to vary problem parameters Should permit "quick-look" visualization


Flash interfacesDriver

runtime_parameterslog_file

performance_monitorphysical_constants

time_dependentinit_block()

evolve()output()

timestep()

steadyinit_block()

solve()output()

tree data structure

init_from_scratch()init_from_checkpoint()

init_flash()

Mesh MaterialsHydro Source_termsGravity

Diffusion

Other

init()fill_guardcells()

test_refinement()refine_derefine()

init()tstep()

set_ext_guardcell()hydro_3d()

init()tstep()

set_ext_guardcell()gravity_3d()

init()tstep()

src_terms()

eos_3d()eos()

abund_renorm()query_iso_db()

Paramesh Uniform Explicit Implicit

Split

PPM

Unsplit

WENO PPM

PPM

constant

point_mass

Poisson

Multigrid

Gamma

Helmholtz

...

burn

coolheat

iso13...

Multipole


F90 Implementation: compile-time "polymorphism"

Config filesPlaintext files describingmodule interrelationships

PARAMETERREQUIRESDEFAULTEXCLUSIVE

Modules filePlaintext files describingmodules to include in code

INCLUDE

Hydroinit()

tstep()set_ext_guardcell()

hydro_3d()Makefile

Explicit

Split

PPMhydro_1d()

etc.Makefile

hydro_3d()hydro_sweep()update_soln()

hydro_1d()

Implicit

Unsplit

WENOhydro_3d()

etc.Makefile

PPMhydro_3d()

etc.Makefile

hydro_3d()

tstep() tstep()

PPMhydro_3d()

etc.Makefile

Setup scriptConfigures build directoryaccording to Modules fileinstructions for a specifiedmachine and problem type

make


Changes from version 1.0

Architectural consistency "Object-oriented" structure

Mimic virtual functions Cumulative inherited Makefiles

Only internal and mesh common blocks Exclusive and non-exclusive modules Test driver capability

Simulation services Runtime parameter contexts Physical constants database (plus unit conversions) Materials properties database Logging and timing libraries

New modules


Automated verification


Flash calculations

Wide range of compressibility Wide range of length and time scales Many interacting physical processes Only indirect validation possible Rapidly evolving computing environment Many people in collaboration

Cellular detonations

Compressible turbulence

Helium burning on neutron starsRayleigh-Taylor instability

Richtmyer-Meshkov instability

Flame-vortex interactions

Laser-driven shock instabilitiesNova outbursts on white dwarfs


Automated verification of Flash

Want verification to be easy and routine Flash includes several test problems

Sod, Sedov, Woodward-Colella blast wave, advection, wind tunnel,

Shu-Osher, isothermal collapse, Jeans instability, … More are added as needed

Automated testing using flash_test Compiles and runs Flash Compares against benchmark results using focu

focu (Flash Output Comparison Utility) Local or global error criteria Generates a short report for flash_test


flash_test

Automated verification architecture

Test databaseRuntime parameterfiles and configurationinformation for alltest problems includedwith code

Site databaseQueue and OSinformation for eachconfigured site; joband job submissiontemplates

OutputLog file indicating success ofeach compile/run/compare;output files

Jobs tracked vialockfile mechanism

Stored benchmark dataIndexed by date; output filescreated by previous invocationof flash_test

Comparisons viafocu


Summary

Flash... Is a small, focused, but flexible framework Offers state-of-the-art simulation capabilities Is extensively verified Achieves high performance (Fryxell talk)

0.24 TFlop/s on ASCI Red (2000 Gordon Bell finalist) Scales to > 6,000 processors

Is producing exciting science (Truran, Dupont talks) Continues to evolve (Siegel talk)

Production Version of Flash : 1.61

Documents

Transcript of Production Version of Flash : 1.61