WRF4G The Weather Research Forecasting model workflow for the GRID [email protected] Department of...
-
Upload
andrew-spencer -
Category
Documents
-
view
216 -
download
3
Transcript of WRF4G The Weather Research Forecasting model workflow for the GRID [email protected] Department of...
WRF4GThe Weather Research Forecasting model workflow for the GRID
[email protected] of Applied Mathematics
& Computer SciencesUniversity of Cantabria, Spain
GRID developers: Valvanuz Fernández, Antonio S. Cofiño
Application developers: Jesús Fernández, Lluís Fita
Santander Meteorology GroupA multidisciplinary approach to weather & climate
http://www.meteo.unican.es
EGU General Assembly 2010, Wien, 2-7 May 2010
A multidisciplinary approach to weather & climateSantander Meteorology Group http://www.meteo.unican.es
Numerical Weather Prediction model (regional)
Open source (Fortran90).
A world community with 6000+ registered users.
Application in a wide variety of studies: Weather forecasting (operational at NCEP)
Data assimilation studies
Regional climate studies
Idealized simulations
Educational applications,....
WRF Model
A multidisciplinary approach to weather & climateSantander Meteorology Group http://www.meteo.unican.es
Computing and storage requirements: Intensive use of CPU
Supports a variety of parallel programming paradigms (OpenMP, MPI, serial,...) . It's a common benchmarking application for HPC.
Large amount of input, output and restart data.
Typical experiments last for days (even in parallel).
Application with a complex workflow (preprocessing, execution & postprocessing)
WRF Requirements
metgrid wrf
geogrid
ungrib
WRF Preprocessing System WRF ARW
WRFGEL
namelist.inputnamelist.wps
real
WR
F W
OR
KF
LOW
A multidisciplinary approach to weather & climateSantander Meteorology Group http://www.meteo.unican.es
Reanalisys/Reforecasting
High number (~104) of independent simulations
High volume of output-data (>TB)
Requires scalability
Regional projections for climate change
Contiguous simulations (~10), weeks of walltime each.
High volume of output data (>10TB)
Recovering system for simulation restart
Weather Forecasting
QoS and optimal resources: deadline for delivering
Sensitive studies for Climate and Weather
Physical schemes, initial conditions and boundary conditions uncertainties sampling
Resource demanding experiments
Scientific experiments WRF
A multidisciplinary approach to weather & climateSantander Meteorology Group http://www.meteo.unican.es
GOALS
Make easier for the user the process of design, execution and monitoring of the experiments with WRF.
Develop a framework that allow the user to use at the same time different computing resources in a transparent way:
Local clusters (PBS, SGE, LoadLeveler,…) Grid infrastructures
(gLite, Globus,…) Local resources
(SSH, fork, …).
Develop a set of command line tools and a Web portal for the WRF users.
WRF4G: Goals
A multidisciplinary approach to weather & climateSantander Meteorology Group http://www.meteo.unican.es
CORDEX (COordinated Regional climate Downscaling EXperiment) is a framework to improve coordination of international efforts in regional climate downscaling research. CORDEX was initiated as a result of the Task Force on Regional Climate Downscaling, formed by the World Climate Research Program (WCRP).
A set of target regions has been proposed and modeling groups willing to contribute must comply with simulations specifications.
CORDEX will produce an ensemble of simulations sampling uncertainties related to: (i) varying Global Climate Model (GCM) simulations; (ii) varying greenhouse gas (GHG) concentration scenarios; (iii) natural climate variability; and (iv) different downscaling methods.
http://wcrp.ipsl.jussieu.fr/RCD_Projects/CORDEXhttp://www.meteo.unican.es/wiki/cordexwrf
There are currently 15 groups planning to contribute to CORDEX with WRF, 5 groups planning to contribute the African domain (key region for the AR5). All of them could benefit from WRF4G app.
Role of WRF4G in CORDEX
A multidisciplinary approach to weather & climateSantander Meteorology Group http://www.meteo.unican.es
150.000 CPUs
70 PB
260 sites worldwide
Arquitectures: i386,x86_64
LRMS: torque, sge, lsf, bqs
Shared and not shared Home
Computing resources: OS: Debian, SL/Centos 4 and 5.
Memory: 250MB to 16GB
Processor: P4 to i7
Different queues limitations: walltime, memory & disk quotas…
Bandwidth in some sites is very small.
EGEE: Example of big Grid infrastructure
A multidisciplinary approach to weather & climateSantander Meteorology Group http://www.meteo.unican.es
GRID Challenges:
Develop an application adapted to run in different arquitectures, OS, LRMS and parallel environments.
Repository of WRF binaries adapted to those environments
Detect sites that not accomplish application requirements: Memory, Processor, queue time limitations,…
Optimizing the data transfers and replication between resources distributed geographically.
Develop a monitoring tool for experiments status.
Create a checkpointing management system that allow to restart simulations.
Failure detection and recovery.
WRF4G: GRID Challenges
A multidisciplinary approach to weather & climateSantander Meteorology Group http://www.meteo.unican.es
WRF4G is a port of the WRF Modeling System prepared to run in GRID environments.
WRF has been splitted in 2 layers to separate the scientific experiment from the infrastructure details (wrf.input & wrf4g.conf).
WRF4G supports several data transfer services (gsiftp, rsync,…) and execution systems (gLite, globus, PBS, SGE,…). It can be run in the user’s machine.
No deployment required on sites. The application is deployed by a pilot job.
WRF for GRID: WRF4G
A multidisciplinary approach to weather & climateSantander Meteorology Group http://www.meteo.unican.es
metgrid real wrf
geogridStatic
GeographicalData
ungribGridded Data:
NAM, GFS, RUC,AGRMET, etc.
WRF Preprocessing System WRF ARW
namelist.inputnamelist.wps
WRF4G Workflow
A multidisciplinary approach to weather & climateSantander Meteorology Group http://www.meteo.unican.es
metgrid real wrf
geogridStatic
GeographicalData
ungribGridded Data:
NAM, GFS, RUC,AGRMET, etc.
WRF Preprocessing System WRF ARWProcessed locally
wrf4g.conf wrf.input
namelist.inputnamelist.wps
WRF4G
WRF4G Workflow
A multidisciplinary approach to weather & climateSantander Meteorology Group http://www.meteo.unican.es
metgrid real wrf
geogridStatic
GeographicalData
ungribGridded Data:
NAM, GFS, RUC,AGRMET, etc.
WRF Preprocessing System WRF ARW
WRFGEL
WRF4G.sh
Processed locally
WRF4G_ini.sh
wrf4g.conf wrf.input
namelist.inputnamelist.wps
wrf4g_submitter.sh
Job submission abstraction layer
WRF4G
UI
WN
Inp
ut
da
ta
ab
stra
ctio
n la
yer
WRF4G Workflow
A multidisciplinary approach to weather & climateSantander Meteorology Group http://www.meteo.unican.es
metgrid real wrf
geogridStatic
GeographicalData
ungribGridded Data:
NAM, GFS, RUC,AGRMET, etc.
WRF Preprocessing System WRF ARW
WRFGEL
WRF4G.sh
Processed locally
WRF4G_ini.sh
wrf4g.conf wrf.input
namelist.inputnamelist.wps
wrf4g_submitter.sh
Job submission abstraction layer
WRF4G
UI
WN
WRF4G-0.0.2.tgzWRF4Gbin-3.1r83.tgz
wrfout_*wrfrst_*
wrfinput_*wrfbdy_*
WRF4G_BASEPATHWRF4G_INPUT
WRF4G_APPS
Inp
ut
da
ta
ab
stra
ctio
n la
yer
geo_em.ncnamelist.wps
WRF4G_DOMAINS
WRF4G Workflow
A multidisciplinary approach to weather & climateSantander Meteorology Group http://www.meteo.unican.es
${WRF4G_ROOT}
scripts
ui wn
WRFGEL WPS WRFV3bin lib
examples
WRF4G_ini.shwrf4g_submitter.shwrf4g_make_tarball.shwrf4g_make_tarball_bin.shwrf4g_submit.EELA_grid_jobwrf4g_submit.MDMclusterwrf4g_submit.MDMclusterIFBwrf4g_wrfgel_environment
create_output_structuredownload_fileexist_wpsget_date_restartregister_file
cccantabria wrf.input wrf4g.confdomain_partition wrf.input.in wrf4g.conf.innino50 wrf.input wrf4g.confoperativo operativo.sh post_operatorio.sh wrf.input.in wrf4g.conf
vcpfortnmlncdumppreprocessor.*
WRF4G.sh
WRF4G structure
A multidisciplinary approach to weather & climateSantander Meteorology Group http://www.meteo.unican.es
These are sample configuration files for a WRF4G experiment
${WRF4G_ROOT}
scripts
ui wn
WRFGEL WPS WRFV3bin lib
examples
WRF4G_ini.shwrf4g_submitter.shwrf4g_make_tarball.shwrf4g_make_tarball_bin.shwrf4g_submit.EELA_grid_jobwrf4g_submit.MDMclusterwrf4g_submit.MDMclusterIFBwrf4g_wrfgel_environment
create_output_structuredownload_fileexist_wpsget_date_restartregister_file
cccantabria wrf.input wrf4g.confdomain_partition wrf.input.in wrf4g.conf.innino50 wrf.input wrf4g.confoperativo operativo.sh post_operatorio.sh wrf.input.in wrf4g.conf
vcpfortnmlncdumppreprocessor.*
WRF4G.sh
WRF4G structure
A multidisciplinary approach to weather & climateSantander Meteorology Group http://www.meteo.unican.es
These are user scripts launched from the UI
${WRF4G_ROOT}
scripts
ui wn
WRFGEL WPS WRFV3bin lib
examples
WRF4G_ini.shwrf4g_submitter.shwrf4g_make_tarball.shwrf4g_make_tarball_bin.shwrf4g_submit.EELA_grid_jobwrf4g_submit.MDMclusterwrf4g_submit.MDMclusterIFBwrf4g_wrfgel_environment
create_output_structuredownload_fileexist_wpsget_date_restartregister_file
cccantabria wrf.input wrf4g.confdomain_partition wrf.input.in wrf4g.conf.innino50 wrf.input wrf4g.confoperativo operativo.sh post_operatorio.sh wrf.input.in wrf4g.conf
vcpfortnmlncdumppreprocessor.*
WRF4G.sh
WRF4G structure
A multidisciplinary approach to weather & climateSantander Meteorology Group http://www.meteo.unican.es
This is what the Worker Node sees just before runningWRF4G.sh
${WRF4G_ROOT}
scripts
ui wn
WRFGEL WPS WRFV3bin lib
examples
WRF4G_ini.shwrf4g_submitter.shwrf4g_make_tarball.shwrf4g_make_tarball_bin.shwrf4g_submit.EELA_grid_jobwrf4g_submit.MDMclusterwrf4g_submit.MDMclusterIFBwrf4g_wrfgel_environment
create_output_structuredownload_fileexist_wpsget_date_restartregister_file
cccantabria wrf.input wrf4g.confdomain_partition wrf.input.in wrf4g.conf.innino50 wrf.input wrf4g.confoperativo operativo.sh post_operatorio.sh wrf.input.in wrf4g.conf
vcpfortnmlncdumppreprocessor.*
WRF4G.sh
WRF4G structure
A multidisciplinary approach to weather & climateSantander Meteorology Group http://www.meteo.unican.es
User Interface - my_experiments/exp
wrf4g.confwrf.input
Create and run an experiment
A multidisciplinary approach to weather & climateSantander Meteorology Group http://www.meteo.unican.es
User Interface - my_experiments/exp
exp__... exp__reaN
0001 0002 ... 0001 0002 ... 0001 0002 ...
wrf4g.confwrf.input
sandbox.tgzwrf.chunkwrf.input
wrf4g.confWRF4G_ini
exp__rea1
realizations
Create and run an experiment
A multidisciplinary approach to weather & climateSantander Meteorology Group http://www.meteo.unican.es
time
Three realizations split into two chunks each:
exp__rea1
exp__rea2
exp__rea3
Storage Element - ${WRF4G_BASEPATH}/experiments/exp
exp__rea1 exp__rea2 exp__rea3
output restart wpsout output restart wpsout output restart wpsout
1 2 30 4 5 6
Create and run an experiment
A multidisciplinary approach to weather & climateSantander Meteorology Group http://www.meteo.unican.es
time
exp__rea1
exp__rea2
exp__rea3
Storage Element - METEO4G/WRF/experiments/exp
exp__rea1 exp__rea2 exp__rea3
output restart wpsout output restart wpsout output restart wpsout
1 2 30 4 5 6
wrfinput_t0wrfbdy_t0
Three realizations split into two chunks each:
Create and run an experiment
A multidisciplinary approach to weather & climateSantander Meteorology Group http://www.meteo.unican.es
time
exp__rea1
exp__rea2
exp__rea3
Storage Element - METEO4G/WRF/experiments/exp
exp__rea1 exp__rea2 exp__rea3
output restart wpsout output restart wpsout output restart wpsout
1 2 30 4 5 6
wrfout_t0wrfout_t1
wrfinput_t0wrfbdy_t0
Three realizations split into two chunks each:
Create and run an experiment
A multidisciplinary approach to weather & climateSantander Meteorology Group http://www.meteo.unican.es
time
exp__rea1
exp__rea2
exp__rea3
Storage Element - METEO4G/WRF/experiments/exp
exp__rea1 exp__rea2 exp__rea3
output restart wpsout output restart wpsout output restart wpsout
1 2 30 4 5 6
wrfout_t0wrfout_t1wrfout_t2 wrfrst_t2
wrfinput_t0wrfbdy_t0
Three realizations split into two chunks each:
Create and run an experiment
A multidisciplinary approach to weather & climateSantander Meteorology Group http://www.meteo.unican.es
time
exp__rea1
exp__rea2
exp__rea3
Storage Element - METEO4G/WRF/experiments/exp
exp__rea1 exp__rea2 exp__rea3
output restart wpsout output restart wpsout output restart wpsout
1 2 30 4 5 6
wrfout_t0wrfout_t1wrfout_t2wrfout_t3wrfout_t4
wrfrst_t2
wrfrst_t4
wrfinput_t0wrfbdy_t0
Three realizations split into two chunks each:
Create and run an experiment
A multidisciplinary approach to weather & climateSantander Meteorology Group http://www.meteo.unican.es
time
exp__rea1
exp__rea2
exp__rea3
Storage Element - METEO4G/WRF/experiments/exp
exp__rea1 exp__rea2 exp__rea3
output restart wpsout output restart wpsout output restart wpsout
1 2 30 4 5 6
wrfout_t0wrfout_t1wrfout_t2wrfout_t3wrfout_t4
wrfrst_t2
wrfrst_t4
wrfinput_t0wrfbdy_t0
wrfinput_t4wrfbdy_t4
Three realizations split into two chunks each:
Create and run an experiment
A multidisciplinary approach to weather & climateSantander Meteorology Group http://www.meteo.unican.es
time
exp__rea1
exp__rea2
exp__rea3
Storage Element - METEO4G/WRF/experiments/exp
exp__rea1 exp__rea2 exp__rea3
output restart wpsout output restart wpsout output restart wpsout
1 2 30 4 5 6
wrfout_t0wrfout_t1wrfout_t2wrfout_t3wrfout_t4
wrfrst_t2
wrfrst_t4
wrfinput_t0wrfbdy_t0
wrfinput_t4wrfbdy_t4
Three realizations split into two chunks each:
Create and run an experiment
A multidisciplinary approach to weather & climateSantander Meteorology Group http://www.meteo.unican.es
time
exp__rea1
exp__rea2
exp__rea3
Storage Element - METEO4G/WRF/experiments/exp
exp__rea1 exp__rea2 exp__rea3
output restart wpsout output restart wpsout output restart wpsout
1 2 30 4 5 6
wrfout_t0wrfout_t1wrfout_t2wrfout_t3wrfout_t4wrfout_t5
wrfrst_t2
wrfrst_t4
wrfinput_t0wrfbdy_t0
wrfinput_t4wrfbdy_t4
Three realizations split into two chunks each:
Create and run an experiment
A multidisciplinary approach to weather & climateSantander Meteorology Group http://www.meteo.unican.es
time
exp__rea1
exp__rea2
exp__rea3
Storage Element - METEO4G/WRF/experiments/exp
exp__rea1 exp__rea2 exp__rea3
output restart wpsout output restart wpsout output restart wpsout
1 2 30 4 5 6
wrfout_t0wrfout_t1wrfout_t2wrfout_t3wrfout_t4wrfout_t5wrfout_t6wrfout_t7wrfout_t8
wrfrst_t2
wrfrst_t4
wrfrst_t6
wrfrst_t8
wrfinput_t0wrfbdy_t0
wrfinput_t4wrfbdy_t4
Three realizations split into two chunks each:
Create and run an experiment
A multidisciplinary approach to weather & climateSantander Meteorology Group http://www.meteo.unican.es
time
exp__rea1
exp__rea2
exp__rea3
Storage Element - METEO4G/WRF/experiments/exp
exp__rea1 exp__rea2 exp__rea3
output restart wpsout output restart wpsout output restart wpsout
wrfout_t0wrfout_t1wrfout_t2wrfout_t3wrfout_t4wrfout_t5wrfout_t6wrfout_t7wrfout_t8
wrfrst_t2
wrfrst_t4
wrfrst_t6
wrfrst_t8
wrfinput_t0wrfbdy_t0
wrfinput_t4wrfbdy_t4
1 2 30 4 5 6
wrfout_t0wrfout_t1wrfout_t2wrfout_t3wrfout_t4wrfout_t5wrfout_t6wrfout_t7wrfout_t8
wrfrst_t2
wrfrst_t4
wrfrst_t6
wrfrst_t8
wrfinput_t0wrfbdy_t0
wrfinput_t4wrfbdy_t4
wrfout_t0wrfout_t1wrfout_t2wrfout_t3wrfout_t4wrfout_t5wrfout_t6wrfout_t7wrfout_t8
wrfrst_t2
wrfrst_t4
wrfrst_t6
wrfrst_t8
wrfinput_t0wrfbdy_t0
wrfinput_t4wrfbdy_t4
Three realizations split into two chunks each:
Create and run an experiment
A multidisciplinary approach to weather & climateSantander Meteorology Group http://www.meteo.unican.es
To hide the complexity of the GRID to the application developer, WRF4G includes several abstraction layers which encapsulate the routine tasks in generic function calls:
Data Management vcp: Provides transparent copy between any of the
following protocols: gsiftp, LFC, rsync, local copies
vcp -r /local/dir gridftp://server:port/remote/pathvcp gsiftp://srv1:port/rmt/file gsiftp://srv2:port/other/filevcp /local/file1 rsync://server/other/local/path/vcp gsiftp://srv1:port/rmt/file ln:link
WRF4G architecture (I)
A multidisciplinary approach to weather & climateSantander Meteorology Group http://www.meteo.unican.es
To hide the complexity of the GRID to the application developer, WRF4G includes several abstraction layers which encapsulate the routine tasks in generic function calls:
Execution Management Through the use of plugins, the user can submit jobs to a
different infrastructures: Grid (globus 4.2 and glite CE), local clusters (PBS, SGE, SLURM,…), local resources.
Prepared to use different execution environments: OpenMP and MPI.
Strong scheduling policy based in history records and resources characteristics.
Failure detection and recovery.
WRF4G architecture (II)
A multidisciplinary approach to weather & climateSantander Meteorology Group http://www.meteo.unican.es
The application is been supported in EELA-2, EGEE & NGI-ES infrastructures and GT2.4 & GT4.
A release candidate (v1.0 RC1) has been launched to run realistic experiments. It consist on a virtual machine UI with all the application components.
Collaborations with end-users in LA (CETA-CIIFEN-
UPS), Europe (CESGA-MeteoGalicia) for feedback and Asia (HAII, Thailand).
Currently we are testing MPI support in the GRID.
Daily operational version running on the GRID for ensemble of weather forecasts.
Conclusions & Current Work
A multidisciplinary approach to weather & climateSantander Meteorology Group http://www.meteo.unican.es
Develop a monitoring system oriented to app status.
Execution management works on Globus and gLite CE based Gridway plugins. PBS y SGE are used, but plugins will developed for better integration with Gridway.
Develop an scalable replica management integrated with Gridway. (bottleneck in GRID infrastructures).
Create a user portal that allow the users managing and monitoring their experiments.
Collaborate with EGI and other International GRID Initiatives like EUAsiaGrid
Incorporate users feedback
Future Work
A multidisciplinary approach to weather & climateSantander Meteorology Group http://www.meteo.unican.es
Thanks [email protected]