MFE Simulation Data Management
description
Transcript of MFE Simulation Data Management
![Page 1: MFE Simulation Data Management](https://reader036.fdocuments.net/reader036/viewer/2022062305/5681591b550346895dc6431b/html5/thumbnails/1.jpg)
MFE Simulation Data Management
SLAC DMW 2004March 16, 2004
W. W. Lee and S. Klasky
Princeton Plasma Physics Laboratory
Princeton, NJ
![Page 2: MFE Simulation Data Management](https://reader036.fdocuments.net/reader036/viewer/2022062305/5681591b550346895dc6431b/html5/thumbnails/2.jpg)
Spatial & Temporal Scales Present Major Challenge to Theory & Simulations
• Huge range of spatial and temporal scales.
• Overlap in scales often means strong (simplified) ordering not possible
• Different codes/theory for different scales.
• 5+years: Integration of physics into Fusion Simulation Project
10-6 10-4 10-2 100 102
Spatial Scales (m)electron gyroradius
Debye length
ion gyroradius
tearing length
skin depth system size
atomic mfp electron-ion mfp
10-10 10-5 100 105
Temporal Scales (s)
electron gyroperiod electron collision
ion gyroperiod Ion collision
inverse electron plasma frequency confinement
Inverse ion plasma frequency current diffusion
pulse length
![Page 3: MFE Simulation Data Management](https://reader036.fdocuments.net/reader036/viewer/2022062305/5681591b550346895dc6431b/html5/thumbnails/3.jpg)
Major Fusion Codes
![Page 4: MFE Simulation Data Management](https://reader036.fdocuments.net/reader036/viewer/2022062305/5681591b550346895dc6431b/html5/thumbnails/4.jpg)
Data Rates of Major Fusion CodesCode (GB)
now / 5yr
Runtimenow/5yr (hr)
Processors
Now/5yr
Mbs
Now/5yr
GTC 4,000 / 100,000 300/150 2048 80/ 1600
Gyro 10 / 100 30/30 512/2048 .8/ 8
GS2 10 / 100 30/30 512/2048 .8 / 8
Degas2 .1 1 10 .2
Transp .05 3 1 .04
Nimrod 5/ 50 20/20 128 .6/ 6
M3D 10 / 100 20/20 128 1.1/ 11
NSTX .25/shot
1/ 40.25 * 40 9, 36
Total (TB) 4.3 / 101
![Page 5: MFE Simulation Data Management](https://reader036.fdocuments.net/reader036/viewer/2022062305/5681591b550346895dc6431b/html5/thumbnails/5.jpg)
Plasma Turbulence Simulation
• Gyrokinetic Particle-In-Cell Simulation -- Reduced Vlasov-Maxwell Equations
• Simulations on MPP Platforms -- Cray T3E & IBM SP (NERSC), Cray-X1 (ORNL), SX6 (Earth Simulator, Japan)
• Simulation of Burning Plasmas -- International Tokamak Experimental Reactor (ITER) • Integrated Fusion Simulation Project
(MFE)• Visualization -- turbulence evolution & particle
orbits
![Page 6: MFE Simulation Data Management](https://reader036.fdocuments.net/reader036/viewer/2022062305/5681591b550346895dc6431b/html5/thumbnails/6.jpg)
Gyrokinetic Approximation
• Gyromotion
• Polarization provides quasineutrality
[W. W. Lee, PF ‘83; JCP ‘87]
![Page 7: MFE Simulation Data Management](https://reader036.fdocuments.net/reader036/viewer/2022062305/5681591b550346895dc6431b/html5/thumbnails/7.jpg)
![Page 8: MFE Simulation Data Management](https://reader036.fdocuments.net/reader036/viewer/2022062305/5681591b550346895dc6431b/html5/thumbnails/8.jpg)
Earth Simulator 18% 10 (Ethier)
![Page 9: MFE Simulation Data Management](https://reader036.fdocuments.net/reader036/viewer/2022062305/5681591b550346895dc6431b/html5/thumbnails/9.jpg)
QuickTime™ and aCinepak decompressor
are needed to see this picture.
QuickTime™ and aVideo decompressor
are needed to see this picture.
Ion Temperature Gradient Driven Turbulence
Electrostatic PotentialParticle Trajectories
![Page 10: MFE Simulation Data Management](https://reader036.fdocuments.net/reader036/viewer/2022062305/5681591b550346895dc6431b/html5/thumbnails/10.jpg)
Data Management challenges• GTC is producing TBs of data
– Data rates: 80Mbs now, 1.6Gbs 5 years.– Need QOS to stream data.
• This data needs to be post-processed– Essential to parallelize the post-processing routines to handle
our larger datasets.– We need a cluster to post process this data.
• M (supercomputer processors) x N (cluster processors) problem.• QOS becomes more important to sustain this post-processing.
• The post-processed data needs to be shared among collaborators– Different sections of the post-processed data may go to different
users .– Post-processed data, along with other metadata should be
archived into a relational database.
![Page 11: MFE Simulation Data Management](https://reader036.fdocuments.net/reader036/viewer/2022062305/5681591b550346895dc6431b/html5/thumbnails/11.jpg)
Post processing of GTC Data.• Particle Data
– No compression possible.– Sent to 1 cluster for visualization/analysis.– Work being done with K. Ma, U.C. Davis: Visualize a million
particles.– Gain new insights into the theory.
• Field Data– Geometric/Temporal compression of the data is possible.– Data needs to be streamed to a local cluster at PPPL.– Reduced subset needs to be sent to PPPL + collaborators.
• Use Logistic Network. [Beck, UT-K]• Data transfer needs to be automatic, and integrated into a
dataflow/webflow for use with parallel analysis routines.
– We desire to see post-processed data during the simulation.
![Page 12: MFE Simulation Data Management](https://reader036.fdocuments.net/reader036/viewer/2022062305/5681591b550346895dc6431b/html5/thumbnails/12.jpg)
After the analysis
• Post-processed data needs to be saved into a relational database– How do we query this abstract data to
compare it with experiments?– 3D correlation functions– Processing of TBs of data/run now, 100’s of
TBs of data/run in 5 years.– Data mining techniques will be necessary to
understand this data.