The Pencil Code -- a high order MPI code for MHD turbulence
Anders Johansen (Sterrewacht Leiden) Axel Brandenburg (NORDITA,
Stockholm) Wolfgang Dobler Tobias Heinemann (DAMTP) Tony Mee
(Newcastle) Wladimir Lyra (Uppsala) etc. (...just google for Pencil
Code)
Slide 2
2 Pencil Code Started in Sept. 2001 by Axel Brandenburg and
Wolfgang Dobler High order (6 th order in space, 3 rd order in
time) Cache & memory efficient MPI, can run PacxMPI (across
countries!) Maintained/developed by many people (CVS!) Automatic
validation (over night or any time) Max resolution so far 1024 3,
256 procs
Slide 3
3 Pencil formulation In CRAY days: worked with full chunks
f(nx,ny,nz,nvar) Now, on SGI, nearly 100% cache misses Instead work
with f(nx,nvar), i.e. one nx-pencil No cache misses, negligible
work space, just 2N Can keep all components of derivative tensors
Communication before sub-timestep Then evaluate all derivatives,
e.g. call curl(f,iA,B) Vector potential A=f(:,:,:,iAx:iAz),
B=B(nx,3)
Slide 4
4 Switch modules magnetic or nomagnetic (e.g. just hydro) hydro
or nohydro (e.g. kinematic dynamo) density or nodensity
(burgulence) entropy or noentropy (e.g. isothermal) radiation or
noradiation (solar convection, discs) dustvelocity or
nodustvelocity (planetesimals) Coagulation, reaction equations
Homochirality (reaction-diffusion-advection equations) Other
physics modules: MHD, radiation, partial ionization, chemical
reactions, self-gravity
Slide 5
5 Pencil Code check-ins
Slide 6
6 High-order schemes Alternative to spectral or compact schemes
Efficiently parallelized, no transpose necessary No restriction on
boundary conditions Curvilinear coordinates possible (except for
singularities) 6th order central differences in space
Non-conservative scheme Allows use of logarithmic density and
entropy Copes well with strong stratification and temperature
contrasts
Slide 7
7 (i) High-order spatial schemes Main advantage: low phase
errors
Slide 8
8 Wavenumber characteristics
Slide 9
9 Higher order less viscosity
Slide 10
10 Less viscosity also in shocks
Slide 11
11 (ii) High-order temporal schemes Main advantage: low
amplitude errors 3 rd order 2 nd order 1 st order 2N-RK3 scheme
(Williamson 1980)
Slide 12
12 Shock tube test
Slide 13
13 Hyperviscous, Smagorinsky, normal Inertial range unaffected
by artificial diffusion Haugen & Brandenburg (PRE,
astro-ph/0402301) height of bottleneck increased onset of
bottleneck at same position
16 Vector potential B=curlA, advantage: divB=0
J=curlB=curl(curlA) =curl2A Not a disadvantage: consider Alfven
waves B-formulation A-formulation 2 nd der once is better than 1 st
der twice!
Slide 17
17 Comparison of A and B methods
Slide 18
18 Wallclock time versus processor # nearly linear Scaling 100
Mb/s shows limitations 1 - 10 Gb/s no limitation
Slide 19
19 Pre-processed data for animations
Slide 20
20 Ma=10 supersonic turbulence
Slide 21
21 Animation of B vectors
Slide 22
22 Animation of energy spectra Very long run at 512 3
resolution
26 Homochirality: competition of left/right Reaction-diffusion
equation
Slide 27
27 Transfer equation & parallelization Analytic Solution:
Ray direction Intrinsic Calculation Processors
Slide 28
28 The Transfer Equation & Parallelization Analytic
Solution: Ray direction Communication Processors
Slide 29
29 The Transfer Equation & Parallelization Analytic
Solution: Ray direction Processors Intrinsic Calculation
Slide 30
30 Current implementation Plasma composed of H and He Only
hydrogen ionization Only H - opacity, calculated analytically No
need for look-up tables Ray directions determined by grid geometry
No interpolation is needed
Slide 31
31 Sunspot with radiative transfer Heinemann et al. 2007
Slide 32
32Conclusions High order schemes Low phase and amplitude errors
Need less viscosity Subgrid scale modeling can be unsafe (some
problems) shallower spectra, longer time scales, different
saturation amplitudes (in helical dynamos) The Pencil Code is
versatile Created for MHD problems, but recent modules include
Radiative transfer and ionization Coagulation equation
Homochirality Global accretion disks Solid particles
Self-gravity...