GYSELA: Turbulent transport in tokamak plasmas - Genci

GYSELA: Turbulent transportin tokamak plasmas

V. Grandgirard1, G. Latu1

Collaborations with physicists:J. Abiteboul2, Y. Dong3, D. Esteve1, X. Garbet1,J.B Girardo1, Ph. Ghendrih1, F. Palermo1,Y. Sarazin1, A. Strugarek7, D. Zarzoso2

Collaborations with mathematicians:A. Back4, T. Cartier-Michaud1, M. Mehrenberger5,L. Mendoza2, E. Sonnendrucker2

Collaborations with computer scientists:J. Bigot6, C. Passeron1,F. Rozar1,6, O. Thomine1

1CEA, IRFM, Cadarache, France2IPP Garching, Germany3LPP, Paris, France4CPT, Marseille, France5IRMA, Strasbourg, France6Maison de la Simulation, Saclay, France7Montreal university, Canada

ANR GYPSI - ANR G8-Exascale NufuseADT-INRIA SELALIB - AEN-INRIA Fusion

, Virginie G N GENCI Gd Challenges N 21 May 2013 1

Turbulence governs Fusion plasma performance

Magnetic ConfinementIn a plasma, particles follow magnetic field lines: Particles may be confined in a toroidal magnetic field

In the sun, plasma is confined by gravityIn a tokamak, plasma is confined by a magnetic field

magnetic toroidal geometry (r , θ, ϕ)

I Certainty: Turbulence limit the maximal value reachable for n and T

à Generate loss of heat and particlesà Confinement properties of the magnetic configuration

I Subject of utmost importance à optimizing experiments like ITER andfuture reactors.


Complex interplay between fields and particles

I Charged particle motion governed by electromagnetic fieldsI Electromagnetic fields governed by charge density ρ and current j

à self-consistent treatment required

à

I Plasma response: The most accurate⇒ Gyrokinetic


Kinetic model for plasma turbulence

I Fields à Maxwell’s equationsI Electrostatic (B = const): E = −∇φ (φ electrostatic potential)I “large scale” (> λDebye)

à Quasi-neutrality equation:

ρ(x, t) =∑

s

nsqs = 0 with ns =

∫fsdv

I Particles à Kinetic approach mandatoryI Fusion plasmas weakly collisional⇒ far from the fluid state

à Boltzmann equation:

∂fs∂t

+ v ·∂fs∂x

+dvdt·∂fs∂v

= C(fs) + S

6D function of s specie fs(x,v) (3D in space and 3D in velocity)


Gyrokinetic theory:à large phase space reduction 6D to 5D

Kinetic theory: à 6D distribution function of particles

I Fusion plasma turbulence is low frequency:ωturb ∼ 105s−1

ωci ∼ 108s−1

I Phase space reduction: fast gyro-motion is averaged outà Adiabatic invariant: magnetic momentum µ = msv2

⊥/(2B)

à Velocity drifts of guiding centers

CEMRACS 2010, Marseille

Transverse driftsTransverse drifts Transverse & parallel dynamics:

Projection on the transverse plane ( ):

(with )

electric drift curvature + ∇B drifts

vG//

vG⊥B, Large reduction memory/CPU time

/ Complexity of the system

Gyrokinetic theory: à 5D distribution function of particlesf(r , θ, ϕ, v‖, µ)


Gyrokinetic codes require state-of-the-art HPC

I GK codes require state-of-the-art HPC techniques and must runefficiently on more than thousands processors.

I non-linear 5D simulationsI multi-scale problem in space and time

I time: ∆t ≈ γ−1∼ 10−6s → tsimul ≈ few τE ∼ 10s

I space: ρi → machine size a ρ∗ ≡ρi

a 1

3 ρ∗ ITER = 1/512

3 Number grid points ∼ (ρ∗)−3

à

Huge mesh for global simulations


GK codes: a highly competitive activity

GYSELA characteristics:

I global : capture large scale events (at the opposite of local codes)I full-f : no scale separation between equilibrium and perturbation as δfI Semi-Lagrangian scheme (= mixed between PIC and eulerian approach)


GYSELA: collaboration with global codes

GYSELA characteristics:

I global : capture large scale events (at the opposite of local codes)I full-f : no scale separation between equilibrium and perturbation as δfI Semi-Lagrangian scheme (= mixed between PIC and eulerian approach)


GYSELA deployed on 9 HPC machines

I GYSELA is actually present on 9 different HPC machines.→ Deployments in 2012-2013: CURIE, HELIOS, CCAMU, Poincare

→ Code adaptation for BlueGene architecture (2012-2013): JUQUEEN, TURING

[Ch. Passeron, G. Latu, J. Bigot (Post-Doc MDS)]

Dedicated to development :- Local cluster : NORMA (256 cores) (2007)- « Mésocentres »: CCAMU-Marseille (~1000 cores) (2012) Maison De la Simulation-Saclay (~1500 cores) (2013) 512 cores dedicated to GYSELA (paid by ANR GYPSI)

Dedicated to development :- Local cluster : NORMA (256 cores) (2007)- « Mésocentres »: CCAMU-Marseille (~1000 cores) (2012) Maison De la Simulation-Saclay (~1500 cores) (2013) 512 cores dedicated to GYSELA (paid by ANR GYPSI)

Dedicated to production : - JADE (CINES Montpellier/France) : ~ 12.000 cores - HPC-FF (Juelich/Germany) : ~ 10.000 cores - TURING (IDRIS Orsay/France) : 65.536 cores

Dedicated to production : - JADE (CINES Montpellier/France) : ~ 12.000 cores - HPC-FF (Juelich/Germany) : ~ 10.000 cores - TURING (IDRIS Orsay/France) : 65.536 cores

Dedicated to production : - CURIE (Saclay/France) : ~ 80.000 cores - IFERC (Rokkasho/Japon) : ~ 80.000 cores - JUQUEEN (Juelich/Germany) : 458.752 cores

Dedicated to production : - CURIE (Saclay/France) : ~ 80.000 cores - IFERC (Rokkasho/Japon) : ~ 80.000 cores - JUQUEEN (Juelich/Germany) : 458.752 cores

HPC pyramid

I HPC-FF and IFERC dedicated to magnetic fusion community


GYSELA CPU time needs à Exponential increase

ITER simulation8 192 cores during 1 month= 6.1 million h. CPU= 6 centuries of computation

~300 billions of points

ITER simulation8 192 cores during 1 month= 6.1 million h. CPU= 6 centuries of computation

~300 billions of points

1 century

2500 x

GYSELA CPU time consumption 25 millions h. CPU16384 cores

20 To of storage

25 millions h. CPU16384 cores

20 To of storage

I GYSELA CPU time needs increase exponentially

I Each Grand Challenge doubles the number of accessible hours per year

I 36 millions of hours for 2013 (PRACE,GENCI,IFERC)à More than 90% dedicated to production runs

: Grand Challenge TURING: 13 millions of hours


GYSELA parallelisation improvements

I Possibility to access to the totality of the machine during “GrandChallenge” sessions give opportunity to test and improve parallelisationof the code

Relative efficiencyNumber of

coresWeak Scaling Strong Scaling

Gd Challenge CINES 2010

92 % 82 % 8192

Gd Challenge CURIE

(mars 2012)91 % 61 % 65536

Porting on Blue Gene Architecture => Communication schemes rewritten

Gd Challenge TURING

(janv 2013)92 % 61 % 65536


WEAK scaling: (on JUQUEEN - Juelich)Relative efficiency of 91% on 458 752 cores

I Parallel communication schemes completely rewrittenI Tests performed on the totality of JUQUEEN/Blue Gene machine (Juelich)

64 128 192 256 320 384 448Nb. of Kcores (x 1000)

0

30

60

90

120

150

180Vlasov solverField solverDerivatives computationDiagnosticsTotal for one run

Execution time, one Gysela (Weak Scaling - Juqueen)

64 128 192 256 320 384 448Nb. of Kcores (x 1000)

0

20

40

60

80

100

120

Vlasov solverField solverDerivatives computationDiagnosticsTotal for one run

Relative efficiency, one run (Weak scaling - Juqueen)

I Weak scaling: Relative efficiency of 91% on 458 752 cores .

I PRACE preparatory access (April 2012 - Nov 2012): 250 000 hoursI ANR G8-Exascale via P. Gibbon.


GYSELA: Biggest simulation ever run

Grand Challenge CINES 2010:

à A simulation close to ITER-size scenario (ρ∗ = 1/512) performed on 1/4torus with additional heating power of 60 MW during 1 ms

ion temperature fluctuationsin the turbulent saturated phase

3 A 5D mesh of 272 109 points(r , θ, ϕ, v‖, µ) = (1024 × 1024 × 128 × 128 × 16)

3 > 6.1 million hours monoproc.I ∼ 31 days on 8192 processors

à

3 6.5 TBytes of data to analyse

I 1.5 TBytes for 2D and 3D savingsI 5 TBytes for restart files

[J. Abiteboul EPS2010, Y. Sarazin IAEA2010]


What’s new in physics

à What has been added since 2010 in terms of physicsI Generation & transport of toroidal rotation / Role of turbulence & boundary

conditionsI [J. Abiteboul et al., PPCF 2013]

I Transport barrier relaxations with Er shearI [A. Strugarek et al., PPCF 2013]I [A. Strugarek et al., PRL submitted]I [Y. Sarazin, V. Grandgirard and A. Strugarek, La Recherche, nov. 2012]

I Interaction energetic particles & turbulence-via EGAMsI [D. Zarzoso et al., PRL 2013]

I Comparison with experimentsI [invited G. Dif-Pradalier , TTF 2013]

I Adding of impurities à Expensive rewritten of the codeI Now one MPI world for speciesI Energy transfer operator & Moment transfer operatorI GC Curie 2012 à Source of impurities is mandatory


ITER-type simulation for a energy confinementtime unreacheable

I Compromise between machine size and simulation up to energy confinementtime must be found

I GYSELA simulation close to ITER-like parameters : 272 billions of pointsI Longest time simulation: 106/Ωc ∼ 1/2 confinement time

Number of

points(*=

i/a)

Time /cNumber of

cores

Number of days

of simulation

Gd ChallengeCINES 2010

272 billions(*=1/512)

147 840 8192 31

Gd ChallengeCURIE 2012

33 billions(*=1/150)

678 510 16384 15

=> Adding of Tritium 32768 4

Comparison with

experiment(in progress)

87 billions(*=1/512)

1 000 000 5520 23


Complex interplay between Energetic particlesdriven modes (EGAMs) & turbulence

Grand Challenge CURIE 2012:

When SEP is on (time> 1800a/cs):

I Slow modification of Fs

I Phase A:Transient Transport Barrier inouter region ρ < 0.5

I Phase B:Saturated EGAMS interact withturbulence

à No regulation of turbulenceby EGAMs

[Zarzoso, PRL 2013 ; Sarazin, IAEA 2012]


Transport barrier dynamics

Grand Challenge CURIE 2012:

I A vorticity source implemented in the codeI Transport Barrier triggered by prescribed E × B sheared flowI The barrier experiences quasi-periodic relaxation events

Snapshots of non-axisymmetricelectric potential fluctuations

[A. Strugarek, La Recherche (2013), PPCF (2012), submitted to PRL (2013)], Virginie G N GENCI Gd Challenges N 21 May 2013 16

GYSELA: On the road towards Exascale

I GYSELA is already using currently Petascale machine

→ Now ITER-like ion simulation: 272 109 points à 6 106 CPU/hours

I GYSELA will require Exascale machine for realistic kinetic electrons

→With electrons: ρions/ρelec = 60à mesh size ×603 and time step/60 !!!


Conclusion - Perspectives

I Each GYSELA simulation = a numerical experiments→ Several weeks on several thousands of core

(ex: Grand Challenge Curie 2012: 15 days on 16384 cores)→ Several TBytes of data to store and to analyse

I Exascale HPC will be required for realistic simulation with both ions andkinetic electrons→ Promising results: Weak scaling - relative efficiency of 91% on 458 752 cores

I Work in progress for Exascale:→ Fault tolerance improvement [O. Thomine] (Post-Doc ANR-G8 Exascale)]

→ Memory reduction per nodes [F. Rozar] (PhD IRFM/ Maison Simulation)

→ Adapting the code for BlueGene architecture [J. Bigot] (Post-Doc MdS)

I Work in progress for WEST:→ Impact of W transport on turbulence [D. Esteve] (PhD)

→ Treatment of realistic geometry [A. Back] (Post-doc ANR GYPSI-CPT)

→ Aligned coordinates (kinetic electrons) [L. Mendoza] (PhD-IPP Garching)


Collaborations:

I ANR GYPSI (2010-2014)→ Strasbourg, Nancy, Marseille

I ANR Nufuse G8@exascale (2012-2016)→ France, Germany, Japan, US, UK

I ADT INRIA Selalib (2011-2015)→ Strasbourg, Bordeaux

I IPL INRIA (march 2013-2017)→ Nice, Bordeaux

I New project following AEN INRIA Fusion(evaluation in progress)→ Strasbourg, Lyon, Nice

I Collaborations with IPP Garching(Germany) since 2012

I Collaborations with “Maison de laSimulation”- Saclay (Paris) since 2012


GYSELA: Turbulent transport in tokamak plasmas - Genci

Documents

Transcript of GYSELA: Turbulent transport in tokamak plasmas - Genci