Post on 26-Jun-2020
Virtual Earthquake and seismology Research Community e-‐science environment in Europe Project 283543 – FP7-‐INFRASTRUCTURES-‐2011-‐2 – www.verce.eu – info@verce.eu
Webinar on the VERCE Portal
Federica Magnoni
10 &17 July 2014
10 & 17 July 2014Webinar on the VERCE Portal
Forward Simula-ons using SPECFEM3D
10 & 17 July 2014Webinar on the VERCE Portal
Spectral Element MethodsPRO CON
Full complexity (topography, 3D tomography, anisotropy, aYenuaZon…)
!Exponential ACCURACY !Computational EFFICIENCY!!!!
(Diagonal Mass Matrix -‐ Explicit time scheme) !Suitable for implementation on parallel architectures !Mechanical proprieties can vary inside each element !Low number of elements per wavelength
!Flexibility
(unstructured, anisotropic, conforming mesh honoring the geology)
HEX
Meshes of Hexahedral Elements
10 & 17 July 2014Webinar on the VERCE Portal
SPECFEM3D_Cartesian (hAps://github.com/geodynamics/specfem3d)
SimulaZons of acousZc (fluid), elasZc (solid), coupled acousZc/elasZc, poroelasZc or seismic wave propagaZon in any type of conforming mesh of hexahedra (structured or not)
!
Can model seismic waves propagaZng in sedimentary basins or any other regional geological model
!
ImplementaZon of free or absorbing surfaces, aYenuaZon, anisotropy, structural heterogeneity (topography, tomography,…), point or finite kinemaZc/dynamic sources
!
Forward, adjoint, noise cross-‐correlaZon simulaZons !
Finite frequency sensiZvity kernels for elasZc/anelasZc and isotropic/anisotropic media
!
Can be also used for exploraZon seismology, non destrucZve tesZng, ocean acousZcs,…
10 & 17 July 2014Webinar on the VERCE Portal
Mesher
SOLVER
ParZZoner SCOTCH or METIS xdecompose_mesh
-‐1
xspecfem3D
CUBIT kernel
CUBIT interface
PYMPI
“GEO
”cub
it
-‐2 xmeshfem3D
Databases xgenerate_databases
Simula-on Strategy
ParZZoner: SCOTCH 5.1.12b (provided with SPECFEM3D)
Par--oning
• Par_file • 10 mesh files from CUBIT
• nproc proc*_Database (binary files)
xdecompose_mesh (serial)
INPUT OUTPUT
Code: SPECFEM3D_Cartesian
xgenerate_databases (parallel)
• nproc proc*_external_mesh.bin (binary files) !
• if ATTENUATION = true nproc proc*_a,enua/on.bin (binary files)
• if SAVE_MESH_FILES = true nproc proc*_[vp,vs,rho,x,y,z,ibool].bin (binary files) nproc proc*_[vp,vs, a,enua/on, res_minimum_period].vtk (ASCII files)
OUTPUT
• Par_file • nproc proc*_Database • velocity model ASCII file
INPUT
Distributed Databases
Code: SPECFEM3D_Cartesian
xgenerate_databases (parallel)
• Par_file • nproc proc*_Database • velocity model ASCII file
INPUT
Distributed Databases
• nproc proc*_external_mesh.bin (binary files) !
• if ATTENUATION = true nproc proc*_a,enua/on.bin (binary files)
• if SAVE_MESH_FILES = true nproc proc*_[vp,vs,rho,x,y,z,ibool].bin (binary files) nproc proc*_[vp,vs, a,enua/on, res_minimum_period].vtk (ASCII files)
OUTPUT
MAIN FLAGS in the Par_file: !
SUPPRESS_UTM_PROJECTION = .false. !UTM_PROJECTION_ZONE = UTM projec/on zone in which your model resides !NPROC = The number of MPI processors !MODEL = default !TOMOGRAPHY_PATH = Directory in which the tomography files are stored !LOCAL_PATH = Directory in which the distributed databases will be wri,en !
!
OpZonal flags in the Par_file: !
SAVE_MESH_FILES = .true. to save ParaView (*.vtk) mesh files !STACEY_ABSORBING_CONDITIONS = .true. to turn on Clayton-‐Enquist absorbing boundary condi/ons !ATTENUATION = .true. if a,enua/on should be incorporated …
Distributed Databases
Distributed Databases
MAIN FLAGS in the Par_file: !
SUPPRESS_UTM_PROJECTION = .false. !UTM_PROJECTION_ZONE = UTM projec/on zone in which your model resides !NPROC = The number of MPI processors !MODEL = default !TOMOGRAPHY_PATH = Directory in which the tomography files are stored !LOCAL_PATH = Directory in which the distributed databases will be wri,en !
!
OpZonal flags in the Par_file: !
SAVE_MESH_FILES = .true. to save ParaView (*.vtk) mesh files !STACEY_ABSORBING_CONDITIONS = .true. to turn on Clayton-‐Enquist absorbing boundary condi/ons !ATTENUATION = .true. if a,enua/on should be incorporated …
Code: SPECFEM3D_Cartesian
xspecfem3D (parallel)
Forward Simula-ons
• Par_file • CMTSOLUTION file for point or finite seismic sources • STATIONS file • nproc proc*_external_mesh.bin (and proc*_a,enua/on.bin if aYenuaZon)
INPUT
• nstat*ncomp seismograms (acc, vel, displ; ASCII files) !• if MOVIE_SURFACE = true
nstep/nstep_between_frames moviedata* (binary files)
OUTPUT
Code: SPECFEM3D_Cartesian
xspecfem3D (parallel)
• Par_file• CMTSOLUTION file for point or finite sources• STATIONS file• nproc proc*_external_mesh.bin (and proc*_attenuation.bin if attenuation)
INPUT
Forward Simula-ons
OUTPUT
• nstat*ncomp seismograms (acc, vel, displ; ASCII files) !• if MOVIE_SURFACE = true
nstep/nstep_between_frames moviedata* (binary files)
OUTPUT
Forward Simula-onsMAIN FLAGS in the Par_file: !
SIMULATION_TYPE = 1 for forward simula/ons !NSTEP = number of /me steps of the simula/on !DT = length of each /me step in seconds !NTSTEP_BETWEEN_OUTPUT_SEISMOS = interval at which synthe/c seismograms are wri,en !
OpZonal flags in the Par_file: !
NTSTEP_BETWEEN_OUTPUT_INFO = interval at which basic informa/on about a run is wri,en to the file system (/mestamp* files) !MOVIE_SURFACE = .true. to create a movie of seismic wave propaga/on on the Earth’s surface !MOVIE_TYPE = 1 to show propaga/on on the top surface !NTSTEP_BETWEEN_FRAMES = number of /mesteps between movie frames !HDUR_MOVIE = half dura/on of the source /me func/on for the movie simula/ons !CREATE_SHAKEMAP = .true. to create a ShakeMap® …
10 & 17 July 2014Webinar on the VERCE Portal
10 & 17 July 2014Webinar on the VERCE Portal
MESH
MATERIAL PROPERTIES (velocity model)
EVENT (INGV, CSEM, GRZ)
STATIONS (ORFEUS)
NUMERICAL CODE (SPECFEM3D)
HPC CONNECTION
HPC CENTER POLICIES
HPC CENTER
SEISMOLOGICAL INGREDIENTS HPC INGREDIENTS
VOLUMETRIC DATAMovieShakemap
SYNTHETIC SEISMOGRAMS
SEISMOLOGICAL OUTPUT
MESH
MATERIAL PROPERTIES (velocity model)
EVENT (INGV, CSEM, GRZ)
STATIONS (ORFEUS)
NUMERICAL CODE (SPECFEM3D)
SEISMOLOGICAL INGREDIENTS
10 & 17 July 2014Webinar on the VERCE Portal
HPC CONNECTION
HPC CENTER POLICIES
HPC CENTER
HPC INGREDIENTS
VOLUMETRIC DATAMovieShakemap
SYNTHETIC SEISMOGRAMS
SEISMOLOGICAL OUTPUT
10 & 17 July 2014Webinar on the VERCE Portal
MESH
MATERIAL PROPERTIES (velocity model)
SEISMOLOGICAL INGREDIENTSUSER MESH+MODEL VALIDATION !
MESH+MODEL LIBRARY
ITALY (5.8 MHex)
C. ITALY (0.2 MHex)
N. ITALY (1.6 MHex)
S. ITALY (1.8 MHex)
FULL ITALY (Di Stefano 2014)
3s 4s
CENTRAL ITALY (Chiarabba 2010)
3s
NORTH ITALY (Di Stefano 2011)
4s
SOUTH ITALY
(Di Stefano 2011)
3sM
esh
Model
10 & 17 July 2014Webinar on the VERCE Portal
21 June 2013 - M 5.2 LUCCASEISMOLOGICAL OUTPUT
BASIC PARAMETERS Zme step 0.003s staZons 114 cores 500 1 minute seismograms !
PERFORMANCE (SuperMUC) input 2 min decomposiZon 8 min solver+movie 14 min seismograms: 8 min stageout: 30 min
NEXT STEPS: what we can do with the portal output right now?
FIRST NEXT STEPS: 3D visualizaZon (volumetric data)
10 & 17 July 2014Webinar on the VERCE Portal
20 May 2012 -‐ M 5.8 EMILIA -‐ finite fault
10 & 17 July 2014Webinar on the VERCE Portal
Tangential Radial Vertical
TEOL
ROVR
PLMA
MAIM
CGRP
SALO
0 30 60 90 1200 30 60 90 120 0 30 60 90 120
ASQU
CRMI
VARE
PESA
MSSA
GORR
Tangential Radial Vertical
0 30 60 90 1200 30 60 90 120 0 30 60 90 120
3.62e-02 cm/s
2.54e-02 cm/s
2.57e-02 cm/s
2.49e-02 cm/s
2.42e-02 cm/s
1.85e-02 cm/s
4.24e-02 cm/s
3.13e-02 cm/s
1.89e-02 cm/s
1.90e-02 cm/s
3.94e-02 cm/s
1.25e-02 cm/s
1D 3D
20 May 2012 -‐ M 5.8 EMILIA
FIRST NEXT STEPS: MT inversion
quality check of models
10 & 17 July 2014Webinar on the VERCE Portal
Available Meshes
ITALY C. ITALY N. ITALY S. ITALY
Num of Hex (Million) 5.8 0.2 1.6 1.8
UTM zone 33 33 32 33
Suggested DT 1E-‐02 5E-‐03 4E-‐03 5E-‐03
CPU Zme (100 cores -‐
60 s)~3 h 25 min ~2 h ~1.5 h
Given a mesh, compuZng Zme scales linearly with: NUM CORES, DT, SIMULATION DURATION
10 & 17 July 2014Webinar on the VERCE Portal
Mesh FilesDESCRIPTION FORMAT
materials_file material associaZons for each element element_ID material_ID
nummaterial_velocity_file material properZes domain_ID negative_material_ID tomography elastic file_name 1 (domain_ID material_ID rho vp vs Qkappa Qmu anisotropy_flag)
nodes_coords_file point locaZons in Cartesian coordinates of the mesh element corners
num_nodes id_node node_coord_x node_coord_y node_coord_z
mesh_file mesh element connecZvity nspec element_ID id_node1 ... id_node8
free_or_absorbing_surface_file_zmaxfree surface connecZvity or surface
connecZvity of the absorbing boundary surface at the top (Zmax)
nspec2D_top id_(element containing the face) id_node1_face .. id_node4_face
absorbing_surface_file_xmax surface connecZvity of the absorbing boundary surface at Xmax
nspec2D_xmax id_(element containing the face) id_node1_face .. id_node4_face
absorbing_surface_file_xmin surface connecZvity of the absorbing boundary surface at Xmin
nspec2D_xmin id_(element containing the face) id_node1_face .. id_node4_face
absorbing_surface_file_ymax surface connecZvity of the absorbing boundary surface at Ymax
nspec2D_ymax id_(element containing the face) id_node1_face .. id_node4_face
absorbing_surface_file_ymin surface connecZvity of the absorbing boundary surface at Ymin
nspec2D_ymin id_(element containing the face) id_node1_face .. id_node4_face
absorbing_surface_file_boAom surface connecZvity of the absorbing boundary surface at the boYom (Zmin)
nspec2D_bottom id_(element containing the face) id_node1_face .. id_node4_face
hYps://github.com/geodynamics/specfem3d/tree/devel/doc/USER_MANUAL
10 & 17 July 2014Webinar on the VERCE Portal
Submit your own Mesh
START SUBMITTING AFTER AUGUST 25 !
Submit the mesh through the portal providing info about: suggested DT (also related to the wave speed model) suggested NPROC UTM zone mesh corners
Fit the available resources: max ~512 cores and run max 2 hours
Produce the 10 mesh files using CUBIT -‐ GEOCUBIT (hYps://github.com/geodynamics/specfem3d/tree/devel/CUBIT_GEOCUBIT)
10 & 17 July 2014Webinar on the VERCE Portal
Available Wave Speed Models
FULL ITALY (for ITALY mesh)
FULL ITALY (for N. ITALY mesh)
CENTRAL ITALY (for C. ITALY mesh)
NORTH ITALY (for N. ITALY mesh)
SOUTH ITALY (for S. ITALY mesh)
UTM zone 33 32 33 32 33
Min Resolved Period (s) 3 4 3 4 3
vp range (m/s) 768 -‐ 8806 804 -‐ 8774 1782 -‐ 8000 1039 -‐ 8609 1732 -‐ 8245
vs range (m/s) 428 -‐ 4725 434 -‐ 4493 1000 -‐ 4444 600 -‐ 4971 1000 -‐ 4761
10 & 17 July 2014Webinar on the VERCE Portal
Tomography File
Produce an ASCII file
START SUBMITTING AFTER AUGUST 25 !
Submit the tomography through the portal providing info about: UTM zone model corners