GEOS–CHEM: Code Management and Benchmarking
description
Transcript of GEOS–CHEM: Code Management and Benchmarking
GEOS–CHEM:Code Management and Benchmarking
Bob YantoscaSoftware EngineerAtmospheric Chemistry Modeling GroupHarvard University
GEOS-CHEM Users’ Meeting04 April 2005
Monday, April 4, 2005 GEOS-CHEM: Code Management and Benchmarking
2
Topics1. Recent history of GEOS–CHEM versions2. Procedure for updating GEOS–CHEM3. Rules and Responsibilities4. GEOS–CHEM 1–Year Benchmarks5. Future directions
6. Appendix: Downloading GEOS–CHEM
Monday, April 4, 2005 GEOS-CHEM: Code Management and Benchmarking
3
Part 1
GEOS–CHEM Recent History
Monday, April 4, 2005 GEOS-CHEM: Code Management and Benchmarking
4
Recent HistoryGEOS–CHEM versions are named vX–YY–ZZ
X is the major version number This is changed when a new GEOS–CHEM version contains
an update that is so revolutionary that it is incompatible with all previous versions.
YY is the the minor version number This is changed each time the model is released publicly
on the GEOS–CHEM website. ZZ is the release number
This is changed for each “internal release”. Very often when we have many updates to add to GEOS–CHEM; these will be added in several stages so that each internal release can be tested separately. Then after all internal releases have been tested, a public release will be made.
Monday, April 4, 2005 GEOS-CHEM: Code Management and Benchmarking
5
Recent HistoryGEOS–CHEM version numbers throughout history:
v1–YY, v2–YY, and v3–YY (all prior to 2000) Ancient history
v4–YY (2000 – 2003) Denotes the ability to run with GEOS–3 met fields
v5–YY–ZZ (2003) Denotes the installation of SMVGEAR II
v6–YY–ZZ (2004) Denotes the ability to run with GEOS–4 met fields
v7–YY–ZZ (Oct 2004 – present) Denotes new and much improved user interface,
updated diagnostics, new emissions, etc…
Monday, April 4, 2005 GEOS-CHEM: Code Management and Benchmarking
6
Recent HistoryIn the last year, GEOS–CHEM development has been particularly rapid. This was mostly driven by:
The need to make G–C compatible with GEOS–4 met fields The need to add science into G–C for the Near-Real-Time
simulations that we would produce for ICARTT
GEOS–4 met fields: Are produced w/ a different GCM than prior GEOS versions Are on a hybrid grid and not a pure-sigma grid Have a different time-stamping scheme than prior GEOS v’s Required substantial recoding of GEOS–CHEM
Monday, April 4, 2005 GEOS-CHEM: Code Management and Benchmarking
7
Recent HistoryUpdates required for GEOS–4 meteorology
Installed new fvDAS TPCORE transport code (B. Field) Mass-flux diagnostics had to be totally rewritten
Installed LLNL pressure-fixer for transport (B. Field) P-fixer “modifies” U & V so that the surface pressure
computed from the winds matches the met field pressure read from disk
Installed new convection code from NCAR (S. Wu) GEOS–4 convection fields are different than GEOS–3 New scheme (from MATCH) includes updrafts & downdrafts
GEOS–4 also required coding changes in several places…
Monday, April 4, 2005 GEOS-CHEM: Code Management and Benchmarking
8
Recent HistoryImportant Scientific Updates to GEOS–CHEM
Addition of aerosol tracers for full-chemistry & offline runs Hydrophilic & hydrophobic BC & OC (R. Park) Secondary organic aerosols (H. Liao & S. Chung, Caltech) Sea salt aerosols (B. Alexander) Dust aerosols (D. Fairlie)
New types of emissions EPA/NEI99 inventory for North America (R. Park & R. Hudman)
NOx, CO, ALK4, ACET, MEK, PRPE, C2H6, C3H8, CH2O, SO2, and SO4
SO2 emissions from ship exhaust (B. Alexander) AVHRR LAI can be turned on or off (M. Fu)
Monday, April 4, 2005 GEOS-CHEM: Code Management and Benchmarking
9
Recent HistoryCoding updates and improvements to GEOS–CHEM
Totally new user interface in v7-yy-zz All GEOS–CHEM switches are now set in one input file
Timeseries diagnostics updated for ICARTT Rewritten for consistency Parallelized with OpenMP for optimal execution
GEOS–CHEM was ported to the SGI/Altix platform Wetdep and cloud convection now parallelize on Altix
Other improvements Production & loss diagnostics were cleaned up Use more allocatable arrays to save memory Removing LOTS of “historical baggage”
Monday, April 4, 2005 GEOS-CHEM: Code Management and Benchmarking
10
Recent HistoryOngoing GEOS–CHEM development …
Mercury simulation (w/ U. Wash) Recoding for MPI parallelization (w/ JPL) Creation of a GEOS–CHEM adjoint (w/ U. Toronto, Caltech) Aerosol microphysics modules (w/ CMU) Interface with CMAQ model (w/ U. Houston, U. Tenn) Online data assimilation (w/ GMAO) GCAP: Interface with GISS–GCM winds (w/ GISS) And many other projects!!
This requires that we have a clearly-defined procedure for updating GEOS–CHEM!!
Monday, April 4, 2005 GEOS-CHEM: Code Management and Benchmarking
11
Part 2
Updating GEOS–CHEM
Monday, April 4, 2005 GEOS-CHEM: Code Management and Benchmarking
12
Updating GEOS–CHEMGEOS–CHEM “standard code” is kept at Harvard
Bob Y. is the “Model Programmer” and maintains the “standard” GEOS–CHEM code
Version control is done with a CVS source code repository Changes from the last released version are documented
All changes to the “standard code” must go through Bob Y. This prevents divergent versions from floating around This also ensures the code will be computationally optimized
Each “standard code” release is benchmarked with a 1-month full-chemistry simulation (more on this later)
For some versions we also run a 1-year benchmark simulation
Monday, April 4, 2005 GEOS-CHEM: Code Management and Benchmarking
13
Updating GEOS–CHEMGEOS–CHEM modification process:
1. Users start by downloading a “standard code” version Available via GEOS–CHEM web site (see Appendix)
2. Users add new science into GEOS–CHEM (and test it!) Please test for 2x2.5, 4x5 (and maybe 1x1) grids Please test for different met fields (e.g. GEOS–3, GEOS–4)
3. When revisions are mature, users submit updates to Bob Y. for inclusion in the next “standard code” release Do not submit unless code is 100% ready to go Please also include any relevant documentation and
required data or other input files
Monday, April 4, 2005 GEOS-CHEM: Code Management and Benchmarking
14
Updating GEOS–CHEMGEOS–CHEM modification process:
4. Bob Y. will add users’ revisions into the “standard code”
5. Bob Y. will test the “standard code” with a 1–month full-chemistry benchmark simulation Benchmark month is 4 x 5 GEOS–3 for July 2001
6. Daniel Jacob must approve the 1-month benchmark results
7. Bob Y. will release the “standard code” and update the documentation on the web site Notification will be made via GEOS–CHEM mailing list
Monday, April 4, 2005 GEOS-CHEM: Code Management and Benchmarking
15
Updating GEOS–CHEM Full chemistry benchmark (July 2001) GEOS-3 meteorology w/ 30 layers @ 4x5 The following types of plots are created:
Tracer ratios w/ previous version (surface, 500hPa) Frequency distribution of tracer and AOD ratios J-Value ratios w/ previous version (surface, 500hPa) Tracer concentrations (surface, 500hPa) and zonal means Budgets for Ox and CO; CH3CCl3 lifetime Emissions table (comparison to previous version) Aerosol optical depths at surface Aerosol optical depth ratios (comparison to previous
version) Results are compared to the previously released
version
Monday, April 4, 2005 GEOS-CHEM: Code Management and Benchmarking
16
Part 3
Rules and Responsibilities
Monday, April 4, 2005 GEOS-CHEM: Code Management and Benchmarking
17
Rules and Responsibilities
GEOS–CHEM user responsibilities Maintain communication with other users Update to new “standard code” versions when
available Do not distribute non-standard code versions Give credit to developers in publications Consult documentation pages on the G–C website Become familiar with debugging techniques Be resourceful! Try to isolate locations in the code
where bugs or errors are encountered before reporting to Bob Y.
Monday, April 4, 2005 GEOS-CHEM: Code Management and Benchmarking
18
Rules and Responsibilities
GEOS–CHEM developer responsibilities Start from the latest available “standard code” release Provide new code for future “standard code” releases
Code should adhere to GEOS–CHEM style guide Provide related data and/or input files:
For 1x1, 2x25 and 4x5 horizontal grids For GEOS–1, GEOS–STRAT, GEOS–3, and GEOS–4 vert. grids Also provide any IDL routines etc. used to create these data
Provide PROFUSE documentation for new code & data files!
Your humble “Model Programmer” isn’t a chemist Think of future users when writing documentation Journal references are very helpful to include in source
code!
Monday, April 4, 2005 GEOS-CHEM: Code Management and Benchmarking
19
Rules and Responsibilities
GEOS–CHEM users and developers should familiarize themselves with the following: Fortran 90
GEOS–CHEM is written in this language
IDL Visualization tools (e.g. GAMAP) are written in IDL netCDF, HDF, HDF–EOS files can be read with IDL
Some shell scripting (e.g. tcsh)
Advanced users: Learn Perl!
Monday, April 4, 2005 GEOS-CHEM: Code Management and Benchmarking
20
Rules and Responsibilities
Why is GEOS–CHEM written in Fortran? Fortran is the de-facto standard for scientific computing
Most existing scientific code is written in Fortran Fortran is much easier to learn than C, C++, Java, etc.
C can be cryptic; Fortran is more English-like Fortran was designed for scientific computation
Fortran has more math functions & data types than C Fortran code optimizes better than C code Most Fortran compilers are OpenMP compatible Fortran executes faster than IDL
Fortran 90 brings many improvements over Fortran 77 You can write structured code with modules
Monday, April 4, 2005 GEOS-CHEM: Code Management and Benchmarking
21
Part 4
1–Year Benchmarks
Monday, April 4, 2005 GEOS-CHEM: Code Management and Benchmarking
22
1–Year Benchmarks 1–month GEOS–CHEM benchmarks are useful for: Tracking changes from one revision to the next Discovering coding errors Ensuring code is ported correctly to new platforms
Shortcomings of 1–month benchmarks 1 month is often not long enough for certain species It is a model-to-model comparison; not model-to-
data
Monday, April 4, 2005 GEOS-CHEM: Code Management and Benchmarking
23
1–Year Benchmarks At the last GEOS–CHEM meeting it was decided to also incorporate a 1–year GEOS–CHEM benchmark GEOS–CHEM versions are compared against
observed data
1–year benchmarks are performed when: A new version of GEOS–CHEM contains a significant
scientific change from a previous version A new set of GMAO met fields is released and we
need to compare how GEOS–CHEM performs w/r/t the previous met fields (e.g. GEOS–4 vs. GEOS–3)
Monday, April 4, 2005 GEOS-CHEM: Code Management and Benchmarking
24
1–Year Benchmarks Description of 1–year benchmark simulations: Coupled NOx–Ox–HC–aerosol simulation Simulation year: 2001 (with 2000 for spin-up) Horizontal Resolution: 4° latitude x 5° longitude Vertical Resolution:
GEOS–3: 30 level pure-sigma grid (lumped from 48 levels)
GEOS–4: 30 level hybrid grid (lumped from 55 levels) Gas-phase chemical tracers:
NOx, Ox, PAN, CO, Lumped >=C4 Alkanes, ISOP, HNO3, H2O2, MEK, Acetaldehyde, Lumped Aldehyde >=C3, MVK, Methacrolein, PMN, PPN, Lumped Alkyl Nitrate, >=C3 Alkenes, C3H8, HCHO, C2H6, N2O5, HNO4, Methyl Hydro Peroxide
Monday, April 4, 2005 GEOS-CHEM: Code Management and Benchmarking
25
1–Year Benchmarks Description of 1–year benchmark simulations: Aerosol tracers:
DMS, SO2, SO4, MSA, NH3, NH4, NIT, Hydrophilic BC, Hydrophilic OC, Hydrophobic BC, Hydrophobic OC, Dust Aerosol (4 size bins), Sea Salt Aerosol (2 size bins)
Secondary Organic Aerosols: ALPH, LIMO, ALCO, SOA1, SOA2, SOA3, SOG1, SOG2, SOG3
Chemical Mechanism: SMVGEAR II (Jacobson et al, 1997), using ~90 chemical
species, ~300 kinetic reactions (including rapid recycling of peroxides as well as H2SO4—HNO3—NH3 chemistry), and 51 photolysis reactions
Photolysis Mechanism: FAST–J algorithm (Wild et al, 2000)
Monday, April 4, 2005 GEOS-CHEM: Code Management and Benchmarking
26
1–Year Benchmarks Description of 1–year benchmark simulations: Operations:
Transport (Lin & Rood, 1996, 1997) Cloud Convection
GEOS–3: (Lin, 1996) GEOS–4: (Rasch, 2001)
Boundary Layer Mixing (Allen, 1996) Emissions Dry Deposition (Jacob et al, 1995) Chemistry
SMVGEAR II + aerosol chemistry schemes Wet Deposition (Jacob et al, 2000)
Monday, April 4, 2005 GEOS-CHEM: Code Management and Benchmarking
27
1–Year Benchmarks Description of 1–year benchmark simulations: Diagnostic output (saved as monthly means):
Optical Depths and Noontime J(O3), J(NO2) Anthropogenic, biomass, and biogenic emission fluxes Dry deposition fluxes and velocities Sources of CO, NOx, and Acetone Concentrations of chemically produced OH, NO, NO2,
HO2 Tracer concentrations Chemical production & loss of Ox and CO Lifetime of Methylchloroform (CH3CCl3) Selected meteorological quantities (U, V, P, T, etc.) Air masses, grid box heights
Monday, April 4, 2005 GEOS-CHEM: Code Management and Benchmarking
28
1–Year Benchmarks Description of 1–year benchmark simulations: The following plots are created:
Model vs. aircraft data (several missions) for: C2H6, C3H8, CO, H2O2, HNO3, NO, O3, PAN
Model CO vs. MOZAIC CO observations Model CO vs. ship track data Model column CO from vs. observations Model CO vs. CMDL CO Model O3 vs. MOZAIC O3 observations Model O3 vs. ozonesonde data Model O3 vs. surface O3 observations
Monday, April 4, 2005 GEOS-CHEM: Code Management and Benchmarking
29
1–Year Benchmarks Results from different 1–year benchmark simulations are usually plotted together against observations
Jennifer Logan is in charge of examining 1–year benchmark results
Jennifer will speak about our most recent 1–year benchmark simulations later today.
Monday, April 4, 2005 GEOS-CHEM: Code Management and Benchmarking
30
Part 5
Future Directions
Monday, April 4, 2005 GEOS-CHEM: Code Management and Benchmarking
31
Future Directions v7–02–04 will be released shortly after this mtg Option to use EPA / NEI 1999 emissions over USA Mercury simulation with online ocean fluxes PBL mixing has been centralized into a single module Planeflight diag. can now read daily files (a la ICARTT) Various bug fixes / improvements
GAMAP v2–03 will also be released shortly Can now read/write to COARDS-compliant netCDF files Other various improvements and minor fixes
Monday, April 4, 2005 GEOS-CHEM: Code Management and Benchmarking
32
Future Directions Earth System Model Framework (ESMF) Is a package of libraries which define “couplings”
between models and/or model components If two models are ESMF compliant then they can
“talk” to each other using a common interface Individual components in the same model may also
connect to each other via ESMF interfaces
GEOS–CHEM is amenable to ESMF compliance It is not ESMF compliant at this moment but… We will move towards ESMF when the need arises GEOS–CHEM coding style does not preclude ESMF
Monday, April 4, 2005 GEOS-CHEM: Code Management and Benchmarking
33
Part 6
Appendix
Monday, April 4, 2005 GEOS-CHEM: Code Management and Benchmarking
34
Downloading GEOS–CHEM
We distribute GEOS–CHEM as a complete software package, with the following components:
Source code and Makefiles Run directories containing user-modifiable input files Data directories containing emissions etc. files Directories containing meteorological field files Web-based documentation Visualization tools (GAMAP)
These components are available via the Internet GEOS–CHEM website FTP server (geos.as.harvard.edu)
Monday, April 4, 2005 GEOS-CHEM: Code Management and Benchmarking
35
Downloading GEOS–CHEM
Source Code Is available from the GEOS-CHEM Website
http://www-as.harvard.edu/chemistry/trop/geos/ Click on Model Versions / Get Code and Data in menu
You must supply the proper password Click on “1-month full chemistry benchmark run
package” This will download GEOS-CHEM.v7-xx-yy.stdrun.tar.gz
Uncompress the file with: gunzip GEOS-CHEM.v7-xx-yy.stdrun.tar.gz
Extract the contents with: tar –xvf GEOS-CHEM.v7-xx-yy.stdrun.tar
Monday, April 4, 2005 GEOS-CHEM: Code Management and Benchmarking
36
Downloading GEOS–CHEM
GEOS-CHEM.v7-xx-yy.stdrun.tar contains Code.v7-xx-yy: Dir w/ GEOS–CHEM source code IDL: Dir w/ IDL routines used to print benchmark results run.fullchem.geos3: Run directory with input files and
1-month benchmark results (more on this later!)
GEOS—CHEM supported compilers SGI MipsPro Linux: Intel (both PC and Altix versions) and PGI HP/Compaq IBM/AIX Sun/SPARC
Monday, April 4, 2005 GEOS-CHEM: Code Management and Benchmarking
37
Downloading GEOS–CHEM
Run directories Are also available from the GEOS-CHEM Website
http://www-as.harvard.edu/chemistry/trop/geos/ Click on Model Versions / Get Code and Data in menu
Supply the user password Click on “4x5 run directories” or “2x25 run
directories” GEOS-CHEM.v7-xx-yy.rundir4x5.tar.gz GEOS-CHEM.v7-xx-yy.rundir2x25.tar.gz
Uncompress the files with: gunzip GEOS-CHEM.v7-xx-yy.rundir4x5.tar.gz, etc…
Extract the contents with: tar –xvf GEOS-CHEM.v7-xx-yy.rundir4x5.tar, etc…
Monday, April 4, 2005 GEOS-CHEM: Code Management and Benchmarking
38
Downloading GEOS–CHEM
GEOS-CHEM.v7-xx-yy.rundir4x5.tar contains: run.geos1.4x5: Dir w/ input files for 4x5 GEOS–1 run.geoss.4x5: Dir w/ input files for 4x5 GEOS–
STRAT run.geos3.4x5: Dir w/ input files for 4x5 GEOS–3 run.geos4.4x5: Dir w/ input files for 4x5 GEOS–4
GEOS-CHEM.v7-xx-yy.rundir2x25.tar contains run.geos1.2x25: Dir w/ input files for 2x2.5 GEOS–1 run.geoss.2x25: Dir w/ input files for 2x2.5 GEOS–S run.geos3.2x25: Dir w/ input files for 2x2.5 GEOS–3 run.geos4.2x25: Dir w/ input files for 2x2.5 GEOS–4
Monday, April 4, 2005 GEOS-CHEM: Code Management and Benchmarking
39
Downloading GEOS–CHEM
Important input files contained in the GEOS–CHEM run directories:
File for setting GEOS–CHEM simulation options: input.geos
For SMVGEAR chemistry mechanism: globchem.dat, mglob.dat, chemga.dat
For FAST-J photolysis: ratj.d, jv_spec.dat, jv_atms.dat
Files containing initial concentrations restart.YYYYMMDDHH (e.g. restart.2001010100)
Monday, April 4, 2005 GEOS-CHEM: Code Management and Benchmarking
40
Downloading GEOS–CHEM
GEOS–CHEM data directories Available via FTP download from geos.as.harvard.edu Contact Jack Yatteau ([email protected]) for access These directories contain “shared” data files (e.g.
emissions, etc.) which are too large to fit in individual users’ space
Data directory structure on geos.as.harvard.edu: /data/GEOS_4x5/: Root-level dir for 4x5 simulation /data/GEOS_2x2.5/: Root-level dir for 2x2.5 simulation /data/GEOS_1x1_CH/: Root-level dir for 1x1 China sim. /data/GEOS_1x1_NA/: Root-level dir for 1x1 N. Am. sim. Data files are grouped into subdirectories by category
Monday, April 4, 2005 GEOS-CHEM: Code Management and Benchmarking
41
Downloading GEOS–CHEM
GEOS–CHEM met field dirs on geos.as.harvard.edu
4x5 met field directories: /data/GEOS_4x5/GEOS_1/YYYY/MM/ /data/GEOS_4x5/GEOS_S/YYYY/MM/ /data/GEOS_4x5/GEOS_3/YYYY/MM/ /data/GEOS_4x5/GEOS_4_v4/YYYY/MM/
2 x 2.5 met field directories: /data/GEOS_4x5/GEOS_1/YYYY/MM/ /data/GEOS_4x5/GEOS_S/YYYY/MM/ /data/GEOS_4x5/GEOS_3/YYYY/MM/ /data/GEOS_4x5/GEOS_4_v4/YYYY/MM/
Monday, April 4, 2005 GEOS-CHEM: Code Management and Benchmarking
42
Downloading GEOS–CHEM
GEOS–CHEM met field dirs on geos.as.harvard.edu
1x1 Nested Grid met field directories: /data/GEOS_1x1_CH/GEOS_3/YYYY/MM/ /data/GEOS_1x1_NA/GEOS_3/YYYY/MM/
NOTES: GEOS–1, GEOS–STRAT, GEOS–3 data files are
gzipped Each month of GEOS–4 data files are stored as a
gzipped TAR file in order to save disk space
Monday, April 4, 2005 GEOS-CHEM: Code Management and Benchmarking
43
Downloading GEOS–CHEM
Visualization tools GAMAP: Global Atmospheric Model Analysis Package
Suite of routines written in IDL Requires IDL version 5.3 or higher
GAMAP routines can be downloaded via the web http://www-as.harvard.edu/chemistry/trop/gamap/ Online documentation is also available at this website
Advantages of using GAMAP Tested on both PC’s and Unix machines; is very stable You can use GAMAP routines with other IDL code IDL has very powerful mapping/imaging capabilities Routines are provided for reading HDF, HDF–EOS,
netCDF files
Monday, April 4, 2005 GEOS-CHEM: Code Management and Benchmarking
44
Downloading GEOS–CHEM
Advantages of using GAMAP GAMAP contains a user-friendly menu-driven interface GAMAP has been tested on PC’s and Unix machines You can combine GAMAP routines with other IDL code
Advantages of using IDL IDL has very powerful mapping/imaging capabilities IDL has routines to read HDF, HDF–EOS, netCDF files You can develop an interactive web interface for
GAMAP with IDL–ION software (sold separately!) See our near-real-time web page (L. Jaegle)
Monday, April 4, 2005 GEOS-CHEM: Code Management and Benchmarking
45
Downloading GEOS–CHEM
Web-based documentation Available through the GEOS–CHEM website
http://www-as.harvard.edu/chemistry/trop/geos/ Click on the Documentation section of the main menu
Types of documentation GEOS–CHEM Frequently-Asked Questions GEOS–CHEM Online Manual GEOS–CHEM Style Guide
Monday, April 4, 2005 GEOS-CHEM: Code Management and Benchmarking
46
Downloading GEOS–CHEM
GEOS–CHEM Frequently-Asked Questions What’s new in GEOS–CHEM? What kinds of simulations can G–C perform? What is the horizontal and vertical resolution of G–C? What kinds of met fields can be used with G–C? What diagnostic quantities can be archived w/ G–C? How is the chemistry mechanism defined in G–C?
Monday, April 4, 2005 GEOS-CHEM: Code Management and Benchmarking
47
Downloading GEOS–CHEM
GEOS–CHEM Online Manual Introduction Installation and compilation Coding: Practice and style
F90 modules Parallelization
Files in the shared data directories Short description of files and their creators References
Files in the run directories Debugging
Monday, April 4, 2005 GEOS-CHEM: Code Management and Benchmarking
48
Downloading GEOS–CHEM
GEOS–CHEM Style Guide Guidelines for writing readable, well-documented code Headers, declarations, indentations, use of white
space New language features of Fortran 90 (including
modules) Fixed-format vs. free-format Numeric and character data types used in GEOS–
CHEM Converting from number to character and vice versa Math optimizations DO Loops