Post on 17-May-2018
Time evolution• 1970: Limited storage (512MB), limited
computing time (12 secs), limited output, no graphics
– develop approximation methods • space-charge kicks, hard-edged magnets • balancing computing time available with
accuracy and number of particles; symplectic v non-symplectic tracking
• core-halo models, analytical modules • frozen space-charge
• Now: Multiprocessors, NERSC, GUIs, 109 macro-particles, jobs running for months, output and store everything we can think of.
2
Why are Codes Important?
3
• Validate theory, give ideas for theoretical and experimental developments • Generate the basic underlying machine design
– sets of self-consistent parameters – optimise machines for performance – avoid resonances, instabilities, minimise non-linear effects
• Establish likely machine performance – predict effect and correction of failure mechanisms – bracketing allowable errors – controlling/reducing beam loss – identify beam properties on exit (e.g. to a target) – quantify output energy, emittance & halo at full current
• Indicate whether novel ideas are feasible • Develop commissioning strategies • Help us avoid making fools of ourselves
The Codes• Beam Optics codes
– Transform envelope with analytical space charge– Used as basis for most tuning algorithms
• PIC Dynamics codes– Linacs: Parmila, Parmela, Tracewin, Dynamion– Rings: Orbit, Simpsons, Simbad, …… new Python/PTC modules– 106 or more particles, with 3-D space charge– Matrix/map based, thin lens+drift+space charge kicks – Do a good job on core simulations– Agree at few % level with some experiments ( perhaps not all)
• Integrating dynamics codes– Impact, Track, Tstep (Parmela), Track2d/3d
• Ray-tracing codes– Zgoubi, G4beamline...
• Can now integrate ~109 particles through field maps
4
Beam Optics Codes v. Beam Tracking
Beam optics codes(example: Trace-3D)
n Matrix based, usually first order n Hard-edge field approximation n Space charge forces approximatedn Beam envelopes and emittancesn Fast, Good for preliminary studiesn Simplex optimisation: Limited number of
fit parameters
Ø Optimisation via optics codes + added terms for specific effectsØ Use beam dynamics codes for checking and final tweaking
– More realistic representation of the beam especially for high-intensity and multiple charge state beams (3D external fields and accurate SC calculation).
– Include quantities not available from beam optics codes: minimise beam halo formation and beam loss.
– Now possible with faster PC’s and parallel computer clusters to use beam dynamics codes for (limited) optimisation.
Beam dynamics codes (example: TRACK, IMPACT)n Particle tracking, all orders includedn 3D fields including realistic fringe fieldsn Solving Poisson equation at every stepn Actual particle distribution: core, halo …n Slower, good for detailed studies including
errors and beam loss n Larger scale optimisation possible
5
Code Limitations
Main issues when modelling real machines:• An accurate 6-D description of the initial beam particle distribution
– beam characterisation, need plenty of diagnostics• Magnets and their alignment can be accurately mapped;• An accurate description of the fields:
– More codes now allow field maps; however, applying/iterating field maps may be an issue
– Some fields are not known:– the axial RF field distribution in RFQ’s is not measurable– the RF field distribution in SC cavities at operating temperature.
• Some diagnostic measurements are not accurate enough to provide the necessary information for the codes
6
Assumptions
• Codes assume particles are distinguishable, moving classically, and separate the forces: space charge, external forces, image charges and currents, beam loading, wake-fields etc.
• Internal forces are separated: space-charge (long-range) and scattering (short-range).
• Assume relative velocities within a bunch are very small, so treating intra-bunch forces as electrostatic - Poisson’s equation.
• Within the limits of the model, we may be confident about tracking particles over short-moderate time scales. But now we are aiming to model full synchrotron cycles, or start-to-end simulations of a whole accelerator complex.
7
Jeff Holmes, NA-PAC 2013
Space Charge• Design codes
– Repeated runs with varying parameters for optimisation, relatively simple – 1D longitudinal codes with space charge from derivative of line density – 2D transverse codes with simple space charge kicks scaled by density
factor – Bunch length >> pipe radius
• Research codes – Detailed “final” runs to explore non-linear effects, resonances, space-
charge induced beam loss etc. – General 3D simulations (essential for short bunches, e.g. linacs) – Space charge from Poisson’s equation using direct coulomb, FFT,
multigrid, fast multipoles, finite differences, finite elements, SOR techniques …
8
Lest we forget…• Consider propagation/growth of errors
– whether a simple symplectic method is better than a more accurate non-symplectic approach
• Programming techniques – computers only do what we tell them to do – computers are not perfect
R. W. Hockney and J. W. Eastwood, “Computer Simulation Using Particles”, Institute of Physics Publishing, Bristol, 1988.
• Codes written for one particular problem/machine may not be suitable for another
– make sure you know what is going on – question results
• In the search for “accuracy”, do not lose sight of the underlying problem.
9
1011 Managed by UT-Battellefor the U.S. Department of Energy
Goodness of Model Depends onInformation Sought
• Individual particle behavior is much more difficult to converge, due to discretization-induced numerical noise. Discretization arises from time-stepping, numerical distribution, and gridding.
Individual particle orbits diffuse in time. Calculated tunes of individual particlesare sensitive to this diffusion.
For long times, convergence criteria are hard to satisfy.
Jeff Holmes, NA-PAC 2013
Simulation with ~109 Particles
With super-fast computers and parallel processors can now simulate a large number of particles: actual number if possible– Suppress noise from the PIC method: enough particles/cell– More detailed simulation: better statistics, better characterisation of beam halo
11
Simulation with ~109 Particles
With super-fast computers and parallel processors can now simulate a large number of particles: actual number if possible– Suppress noise from the PIC method: enough particles/cell– More detailed simulation: better statistics, better characterisation of beam halo
11
Phase space plotsfor 8.65×108 protonsafter 30 cells in the SNS RFQ.
Longitudinal Tracking of the SNS RFQ
Simulation with ~109 Particles
With super-fast computers and parallel processors can now simulate a large number of particles: actual number if possible– Suppress noise from the PIC method: enough particles/cell– More detailed simulation: better statistics, better characterisation of beam halo
11
TRACK, 109 particle simulation SNS measurement in MEBT
Courtesy: Mustapha, ANL Courtesy: Jeon, SNS
Even 1 billion particles may not provide enough detail
Rings Codes: Inventory• Single particle transport through various types of lattice elements as well as
sufficiently many particles stop study collective effects• Magnet Errors, Closed Orbit Calculation, Orbit Correction• Charge exchange injection foil and phase space painting• RF and acceleration• Longitudinal impedance and 1D longitudinal space charge• Transverse impedance• 2.5D space charge with or without conducting wall beam pipe• 3D space charge• Field maps• Feedback for Stabilisation• Apertures and collimation• Electron Cloud Model
12
Code Language Platform GUI Parallel 1D/2D/3D Particles linacs/rings
IMPACT F90 Unix/Linux no MPI 3D > 106 linacs
ML-IMPACT F90 Unix/Linux/Mac no MPI 3D > 106 linacs/rings
PARMILA F90 Windows no no 2D/3D 104-105 linacs/transfer lines
GPT C, C++ Windows yes MPI scans 3D 106 linacs/FEL/transfer lines
BEST F90 Unix/Linux python/IDL MPI/ 3D > 106 linacs/ringsOpenMP
VADOR C++ Unix/Linux no MPI 2D n/a linacs
SPUNCH F77 Linux no 1D 104 LEBT
PATH F90 Windows yes no 3D 105 linacs/rings
TRACEWIN C++ Windows yes no 2D/3D 105 linacs
DYNAC F77 Linux/Unix/ no no 2D/3D 105 linacsWindows
Synergia F90/C++/ Unix no MPI 3D > 106 linacs/ringsPython
WARP Python/ Linux/Unix/ Under dev MPI 3D/rz/xy up to 108 linacs/ringsF77/F90/C Windows/Mac
Spreadsheet of Space-Charge Codes I
13
Mac, Windows
Mac
>
Spreadsheet of Space-Charge Codes IICode Space Charge Solver Boundaries/Images Impedances Field Maps Integration
order
IMPACT spectral open/periodic/ no yes 2nd order in zrectangular/circular
ML-IMPACT spectral elliptical/ yes no 2nd in zpolygon/lossy 5th Runge-Kutta
PARMILA
GPT 3D multigrid open conductive rect. no 2D,3D 5th Runge-Kuttapipe, cathode
BEST spectral, FD circular conducting wall automatic/ no user specifiedexternal
VADOR FFT conductive wall no no 2ndany shape
SPUNCH exact for disc- circular conducting n/ n/a 1stshaped particles wall
PATH Schell, pt-to-pt open no yes ?
TRACEWIN Sche↵/PICNIC/Gaussup open no no ?
DYNAC Sche↵/Scherm/Hersc open no yes 3rd analytical
Synergia spectral (IMPACT) open/periodic/ no yes 2nd order in zrectangular/circular
WARP FFT, Cap matrix, square/round pipe, ad hoc no 2nd ordermultigrid, adaptive mesh, internal conductors,
refined MG bent pipe, general
14
Yes
Spreadsheet of Space-Charge Codes III
Code t or s tracking Graphics Portability Source code Manual Standard
available? test cases
IMPACT s post proc. all unix platforms to collaborators partial yeswith MPI
ML-IMPACT s post proc. to collaborators partial yes
GPT t built in portable except all beamline yes yesuser interface components
BEST t netcdf, IDL any Linux yes no yes
VADOR s GNUplot, any Linux yes almost yesopenDX
SPUNCH s built in fully portable yes no yes
PATH s built in any Windows yes yes ?
TRACEWIN s built in any Windows no yes ?
DYNAC t, s GNUplot fully portable yes yes yes
Synergia s post proc. all Linux yes in progress not yetRoot+Openinventor
WARP t, s PyGist 2D portable yes online yesOpenDX 3D
15
Issues• How valid are the initial approach and initial assumptions in
PIC-type modelling? • How far can we trust and how useful are codes based on
frozen space charge? • Can we believe results from extended modelling runs (weeks
or more)? – symplectic v. non-symplectic – integrated methods, drift/matrix+space charge kicks
16
RIAPMTQ The Fortran 90 version of PARMTEQM was the basis
for RIAPMTQ. The code was “parallelized” by
incorporating Message Passing Interface (MPI)
commands to allow the code to run in the parallel-multi-
processor environment. Optimization of the code using
“domain decomposition” was not thought to be necessary
for this initial phase; the simpler approach called “particle
decomposition” was used. The most significant code
modifications were required in the parallelization of the
space-charge calculations. The following modifications
were made to RIAPMTQ: transport and acceleration of
multiple-charge-state beams (two at present before the
first stripper), RFQ input transition cell which is desirable
when the RFQ begins with vane modulations, beam-line
elements including high-voltage platforms within the
linac, interdigital accelerating structures, charge-stripper
foils, capabilities for simulations of the effects of machine
errors including misalignments and other off-normal
operating conditions, and automated beam steering where
the program applies the steering that is needed. We have
both a PC version of the code and a parallel-processor
version.
IMPACT The IMPACT code is a parallel particle-in-cell (PIC)
beam dynamics code. It has a large collection of beam
line elements, calculates the acceleration numerically
using RF cavity fields obtained from electromagnetic
field-solver codes, and calculates 3D space charge with
several boundary conditions. The IMPACT code
modifications for a heavy-ion driver linac included
multiple-charge-state capability, improved modelling of
bending magnets, stripping models, a beam scraper, and a
multipole magnet model including a sextupole, octupole,
and decapole.
MULTIPROCESSOR END-TO-END SIMULATIONS
RIAPMTQ/IMPACT end-to-end simulations using
parallel processing were performed at the NERSC facility
at LBNL for both the ANL and MSU RIA driver-linac
designs using 10 million particles, each beginning at the
multiharmonic buncher in the LEBT with transverse 4-D
waterbag distributions that are uniformly distributed in
phase. The MSU and the ANL simulations were done
using 16 and 32 processors, respectively. The IMPACT
simulations from the end of the MEBT to the end of the
linac included charge stripping and beam selection
sections. Figures 1 and 2 show 10 million simulation-
particle final transverse phase-space results for the MSU
(uranium charge states q=87, 88, and 80) and ANL
(uranium charge states q=86, 87, 88, 89, and 90) RIA
driver linac designs, respectively.
Rather than plot all 10M particles we employ a
technique in which we impose a threshold cut in the 2D
projections: in regions of the 2D phase space where the
projected density is below the threshold, we plot every
simulation particle; in regions where the projected density
is above threshold, we use a rejection method to
statistically sample the particles to approximately hold the
plotted density at the threshold value. This allows us to
completely observe the halo without plotting a needlessly
large number of particles in the beam core. For the phase
space plots below, the threshold was taken to be 0.1% of
the maximum density in each projection. For illustrative
purposes the plots also contain the particles comprising
every 100th
particle of the distribution.
particle simulation for three-charge-state uranium beam at
the exit of linac for the MSU RIA design. As described in
the text, the plotting algorithm displays all the halo
particles. The dense central region is obtained by plotting
every 100th
particle.
particle simulation for five-charge-state uranium beam at
the exit of linac for the ANL RIA design. As described in
the text, the plotting algorithm displays all the halo
particles. The dense central region is obtained by plotting
every 100th
particle.
Figure 1: Transverse phase-space plot (x vs px) of 10M
Figure 2: Transverse phase-space plot (x vs px) of 10M
Proceedings of PAC07, Albuquerque, New Mexico, USA THPAS051
05 Beam Dynamics and Electromagnetic Fields
1-4244-0917-9/07/$25.00 c�2007 IEEE
D05 Code Developments and Simulation Techniques
3607
Codes discussion this afternoon (S. Machida)
Transverse phase-space plot (x vs px) of 10M particle simulation for five-charge-state uranium beam at the exit of linac for the ANL RIA design