Transcript of The Pencil Code -- a high order MPI code for MHD turbulence Anders Johansen (Sterrewacht Leiden)...
- Slide 1
- The Pencil Code -- a high order MPI code for MHD turbulence
Anders Johansen (Sterrewacht Leiden) Axel Brandenburg (NORDITA,
Stockholm) Wolfgang Dobler Tobias Heinemann (DAMTP) Tony Mee
(Newcastle) Wladimir Lyra (Uppsala) etc. (...just google for Pencil
Code)
- Slide 2
- 2 Pencil Code Started in Sept. 2001 by Axel Brandenburg and
Wolfgang Dobler High order (6 th order in space, 3 rd order in
time) Cache & memory efficient MPI, can run PacxMPI (across
countries!) Maintained/developed by many people (CVS!) Automatic
validation (over night or any time) Max resolution so far 1024 3,
256 procs
- Slide 3
- 3 Pencil formulation In CRAY days: worked with full chunks
f(nx,ny,nz,nvar) Now, on SGI, nearly 100% cache misses Instead work
with f(nx,nvar), i.e. one nx-pencil No cache misses, negligible
work space, just 2N Can keep all components of derivative tensors
Communication before sub-timestep Then evaluate all derivatives,
e.g. call curl(f,iA,B) Vector potential A=f(:,:,:,iAx:iAz),
B=B(nx,3)
- Slide 4
- 4 Switch modules magnetic or nomagnetic (e.g. just hydro) hydro
or nohydro (e.g. kinematic dynamo) density or nodensity
(burgulence) entropy or noentropy (e.g. isothermal) radiation or
noradiation (solar convection, discs) dustvelocity or
nodustvelocity (planetesimals) Coagulation, reaction equations
Homochirality (reaction-diffusion-advection equations) Other
physics modules: MHD, radiation, partial ionization, chemical
reactions, self-gravity
- Slide 5
- 5 Pencil Code check-ins
- Slide 6
- 6 High-order schemes Alternative to spectral or compact schemes
Efficiently parallelized, no transpose necessary No restriction on
boundary conditions Curvilinear coordinates possible (except for
singularities) 6th order central differences in space
Non-conservative scheme Allows use of logarithmic density and
entropy Copes well with strong stratification and temperature
contrasts
- Slide 7
- 7 (i) High-order spatial schemes Main advantage: low phase
errors
- Slide 8
- 8 Wavenumber characteristics
- Slide 9
- 9 Higher order less viscosity
- Slide 10
- 10 Less viscosity also in shocks
- Slide 11
- 11 (ii) High-order temporal schemes Main advantage: low
amplitude errors 3 rd order 2 nd order 1 st order 2N-RK3 scheme
(Williamson 1980)
- Slide 12
- 12 Shock tube test
- Slide 13
- 13 Hyperviscous, Smagorinsky, normal Inertial range unaffected
by artificial diffusion Haugen & Brandenburg (PRE,
astro-ph/0402301) height of bottleneck increased onset of
bottleneck at same position
- Slide 14
- 14 256 processor run at 1024 3
- Slide 15
- 15 MHD equations Induction Equation: Magn. Vector potential
Momentum and Continuity eqns
- Slide 16
- 16 Vector potential B=curlA, advantage: divB=0
J=curlB=curl(curlA) =curl2A Not a disadvantage: consider Alfven
waves B-formulation A-formulation 2 nd der once is better than 1 st
der twice!
- Slide 17
- 17 Comparison of A and B methods
- Slide 18
- 18 Wallclock time versus processor # nearly linear Scaling 100
Mb/s shows limitations 1 - 10 Gb/s no limitation
- Slide 19
- 19 Pre-processed data for animations
- Slide 20
- 20 Ma=10 supersonic turbulence
- Slide 21
- 21 Animation of B vectors
- Slide 22
- 22 Animation of energy spectra Very long run at 512 3
resolution
- Slide 23
- 23 MRI turbulence MRI = magnetorotational instability 256 3 w/o
hypervisc. t = 600 = 20 orbits 512 3 w/o hypervisc. t = 60 = 2
orbits
- Slide 24
- 24 Fully convective star
- Slide 25
- 25 Geodynamo simulation
- Slide 26
- 26 Homochirality: competition of left/right Reaction-diffusion
equation
- Slide 27
- 27 Transfer equation & parallelization Analytic Solution:
Ray direction Intrinsic Calculation Processors
- Slide 28
- 28 The Transfer Equation & Parallelization Analytic
Solution: Ray direction Communication Processors
- Slide 29
- 29 The Transfer Equation & Parallelization Analytic
Solution: Ray direction Processors Intrinsic Calculation
- Slide 30
- 30 Current implementation Plasma composed of H and He Only
hydrogen ionization Only H - opacity, calculated analytically No
need for look-up tables Ray directions determined by grid geometry
No interpolation is needed
- Slide 31
- 31 Sunspot with radiative transfer Heinemann et al. 2007
- Slide 32
- 32Conclusions High order schemes Low phase and amplitude errors
Need less viscosity Subgrid scale modeling can be unsafe (some
problems) shallower spectra, longer time scales, different
saturation amplitudes (in helical dynamos) The Pencil Code is
versatile Created for MHD problems, but recent modules include
Radiative transfer and ionization Coagulation equation
Homochirality Global accretion disks Solid particles
Self-gravity...