DOE/HEP SciDAC AST Project:

“Advanced Computing for 21st Century Accelerator Science and Technology”

Impact of SciDAC on Accelerator Projects Across SC through Electromagnetic Modeling

Kwok Ko

Stanford Linear Accelerator Center

* Work supported by U.S. DOE ASCR & HEP Divisions under contract DE-AC02-76SF00515

Rich Lee - Achievements in ISIC/SAPP Collaborations for Electromagnetic Modeling of Accelerators (Poster)

DOE Office of Science Accelerators

LCLS

PEP-II

HEPILC

NP

CEBAF

Accelerators are essential tools for doing science in SC - close to 50% of the Facilities for the Future of Science involve accelerators

BES

LCLS

PEP II

ILC

RIA

ILCLCLS

Electromagnetic Structures in Accelerators

High resolution modeling and end-to-end simulation are being used to improve existing accelerators (e.g. PEP-II) and design future facilities (e.g. ILC).

Due to the large complex geometry and required accuracy Large-scale computation is absolutely essential

FEL

Electromagnetic Modeling

NLC Cell Design

CAD/Meshing Partitioning Solvers Analysis

SciDAC has enabled collaborations with the ISICs and SAPP to advance the computational science needed for Large-scale electromagnetic modeling .

Refinement Optimization VisualizationPerformance

AST- Electromagnetics Project

Simulation and Modeling

ComputationalScience

Parallel Code Development

HHigh Performance Computing (NERSC, ORNL)

Accelerators

SLACDESYKEKJlabANLMITPSI

SLAC

Accelerator Modeling

Computational Mathematics

Computing Technologies

ISICs/SAPP

LBNLLLNLSNLStanford, UCDRPI, CMUColumbiaUWisconsin

SciDAC

UCD

K. Ma, H. Yu Z. Bai

AST Electromagnetics Team

RPI

M. Shephard, A. Brewer, E. Seol

SNL

P. Knupp, K. Devine.L. Fisk, J. Kraftcheck

LBNL

E. Ng, W. Gao, X. Li, C, YangP. Husbands, A. Pinar,D. Bailey, D. Gunter

LLNL

L. Diachin, D. Brown, D. Quinlan, R. Vuduc

ISICs (TSTT, TOPS, PERC) and SAPP

Columbia

D. Keyes

CMU

O. Ghattas V. Akcelik

Computational Mathematics

L. Lee, L. Ge, E. Prudencio, S. Chen (Stanford),

Accelerator Modeling

K. Ko, V. Ivanov, A. Kabel, Z. Li, C. Ng,, L. Xiao, A. Candel (PSI)

Computing Technologies

N. Folwell, G. Schussman, R. Uplenchwar, A. Guetz (Stanford)

SLAC/ACD

UWisconsin

T. Tautges, H. Kim,

Parallel EM Code Development

Solve Maxwell’s equations in time & frequency domains using unstructured grid and parallel computingGeneralizedYee Grid

Omega3PTau3P/T3P S3P

Time Domain Simulation

With Excitations

Frequency DomainMode Calculation

Scattering Matrix Evaluation

Finite-Element Discretization

Track3P – Particle Tracking with Surface Physics

V3D – Visualization/Animation of Meshes, Particles & Fields

Achievements in Accelerator Science/Design

PEP-II - IR Heating - Vertex Bellows Heating NLC – DDS Cell Design - DDS Wakefields - Dark Current Pulse RIA - RFQ Design and AMR LCLS - RF Gun Design

HEP PEP-II - IR Heating

Omega3P/Tau3P calculated heat power distribution to help in the IR upgrade design to reduce beam heating

TSTT generated good quality hexahedral meshes to enable Tau3P simulation with beam

Power Distribution

Total power = 17.2 kW

for 3A (330 Modes)

15% higher beam current since upgrade led to higher luminosity/physics discovery

PEP-IIIR

Omega3P was used to evaluate the damping of localized modes by mounting ceramic tiles on the bellows convolution. Bellows modes were found to be damped to very low Qs

Bellows Modes

Bellows mode Ceramic tile absorber Dielectric loss

HEP PEP-II – Vertex Bellows Heating

HEP NLC - DDS Cell Design

-2

-1.5

-1

-0.5

0

0.5

1

1.5

2

0 50 100 150 200

Single-disk RF-QCdel_sf00del_sf0pidel_sf1pidel_sf20

Frequency Deviation [MHz]

Disk number

+1MHz

-1MHz

Omega3P provided the dimensions for 206 NLC Damped Detuned Structure cells. Microwave QC of cells verified frequency accuracy of 1 part in 10,000 as targeted.

SLAC/Stanford/LBL (SAPP/TOPS) developed faster and more scalable eigensolvers (ISIL & ESIL)

Potential savings of $100 million+ in NLC machine cost since DDS is 14% more efficient than standard cell design

New core accelerator design capability

Omega3P: Sum over eigenmodes

Omega3P/Tau3P computed the long-range wakefields in the entire 55-cell DDS to assess the NLC baseline design in wakefield suppression.

SLAC/SNL/LBL (SAPP) improved Tau3P speedup with alternate partitioning tools from Zoltan

HEP NLC - DDS Wakefields

Tau3P: Direct beam excitation

HEP NLC - Wakefields Comparison

Omega3P

Tau3P

Mode Spectrum Wakefields behind leading bunch

1st ever wakefield analysis of an actual DDS prototype Provided benchmarking of Omega3P and Tau3P

Dark current @ 3 pulse risetimes

Track3P Data

-- 10 nsec-- 15 nsec-- 20 nsec

Dark current pulses were simulated for the 1st time in a 30-cell X-band structure with Track3P and compared with data. Simulation shows increase in dark current during pulse risetime due to field enhancement from dispersive effects.

Track3P: Dark current simulation

Red – Primary particles, Green – Secondary particles Red – Primary particles, Green – Secondary particles

Track3P: Dark current simulation

HEP NLC – Dark Current Pulse

NP RIA - RFQ DesignOmega3P was used to model RIA’s low energy RFQ. With Adaptive Meshing Refinement accuracy in frequency and wall loss calculations improved by a factor of 10 and 2 respectively while using a fraction of CPU time required for no AMR case.

SLAC\RPI (TSTT) developed AMR in Omega3P to speed up convergence, improve accuracy and reduce computing time.

Wall Loss on AMR Mesh

RFQ

RFQ - Frequency Convergence

54.354.454.554.654.754.854.9

5555.155.2

0 1000000 2000000 3000000 4000000

Number of Unknowns

Frequency in MHz

RFQ − Q Convergence

5750

5800

5850

5900

5950

6000

6050

6100

0 1000000 2000000 3000000 4000000Number of Unknowns

Q

FrequencyConvergence

Qo Convergence

More accurate f and Q predictions reduce the number of tuners and tuning range, and allow for better cooling design

Omega3P/S3P provided the dimensions for the LCLS RF Gun cavity that meet two important requirements:

minimized dipole and quadrupole fields via a racetrack dual-feed coupler design,

reduced pulse heating by rounding of the z coupling iris.

-0.003

-0.002

-0.001

0.000

0.001

0.002

0.003

-200 -100 0 100 200

rf phase (degree)

cylindrical cavity

racetrack with offset=0.05 "

Qu

ad

(

βr)

/mm

A new parallel Particle-In-Cell (PIC) capability is being developed in T3P for self-consistent modeling of RF guns needed for the LCLS upgrade, future light sources and FELs.

Quad

BES LCLS – RF Gun Cavity

Projects in Progress

ILC – Cavity Design Eigensolvers Visualization Shape Optimization Parallel Meshing Performance

HEP ILC – Accelerating Cavity

The ILC, highest priority of future HEP projects, will be the most costly accelerator (many billion $) and is being designed by an international team Europe, Asia and North America.

An international collaboration (KEK, DESY, SLAC, FNAL and Jlab) is assessing a Low-Loss (LL) design as a viable option for the ILC superconducting RF accelerating cavity.

HEP ILC – Low-Loss Cavity Design

The LL Cavity uses a cavity shape that has 23% less cryogenic loss. The design challenge is to ensure damping of the HOMs over a broad band of frequencies via the two HOM couplers while maintaining the efficiency of the fundamental mode.

Two variants of the LL cavity are being considered, the DESY LL design and the KEK Ichiro design. SLAC is simulating both designs.

Partitioned Mesh of LL Cavity

Qext

With new advances in eigensolvers under SciDAC, Omega3P can now compute the complex frequency or Qe = r / i of HOMs as a result of damping by the HOM couplers

HEP ILC – HOM Damping

Qext in ICHIRO-2 cavity

1.E+01

1.E+02

1.E+03

1.E+04

1.E+05

1.E+06

1.45E+09 1.55E+09 1.65E+09 1.75E+09 1.85E+09 1.95E+09

F (GHz)

Qext

DESY KEK

Qe Qe

Advances in Eigensolvers

i

Omega3P

Lossless Lossy Material

PeriodicStructure

ExternalCoupling

ESILISIL w/ refinement

Implicit RestartedArnoldi SOAR Self-Consistent

Loop

WSMP MUMPS SuperLU Krylov Subspace Methods

Domain-specific preconditioners

Cavity designs with coupling to external waveguides require solutions to a nonlinear eigenvalue problem as the boundary conditions also depend on the eigenvalue. Two solvers, SOAR and SCL have been applied successfully to the ILC cavities.

(SLAC, TOPS/SAPP - LBL, Stanford, UC Davis)

Advances in Visualization

New graphics tools for rendering LARGE, multi-stream, 3D unstructured data have been developed, to be supported by a dedicated visualization cluster to help in analyzing cavity design, such as mode rotation in the ILC cavity.

Rendering of LARGE, 3D multi-stream, unstructured data is essential to the analysis of accelerators, such as mode rotation exhibited by the two polarizations of a damped mode in the ILC cavity as they are coupled through resonance overlap.

(SLAC, SAPP - UC Davis)

R

b

b1a1

La2

b2

Advances in Shape Optimization

A parallel shape optimization capability is under development in Omega3P for optimizing the ILC cavity end cells to damp trapped HOMs so the cavity can meet beam stability requirements.

Omega3PSensitivity

Optimizer Geometricmodel

Omega3P Meshing

(SLAC, TOPS – CMU, LBL, Columbia, TSTT – SNL, LLNL)

Advances in Meshing

To model a chain of cavities within a cryomodule a parallel meshing capability has been developed to overcome the single CPU memory limitation of standard meshing software.

Processor: 1 2 3 4

4 cavity cryomodule at STF (KEK)

(SLAC, TSTT – U Wisconsin, SNL)

CAD-based Partitioning for Parallel Meshing

Advances in Omega3P Performance

(SLAC, PERC – LBL, LLNL)Breakdown of Solve & Postprocess

Speedup after code optimization

LLNL

Latest communication pattern study on NERSC

LBL

AST- ESS under SciDAC Support Accelerator Science/Design across SC Advance Computational Science through ISICs and SAPP

BPM & Wakefields in LCLS Undulator

ILC

XFEL SC RF Gun MIT PBG

Cavity for Jlab 12 GeV Upgrade

ILC LL Cavity & Cryomodule