Bob Yantosca Software Engineer

Bob YantoscaSoftware Engineer

Atmospheric Chemistry Modeling GroupSchool of Engineering & Applied Sciences

Harvard University

GEOS–Chem Updateand a look ahead to new things

Jacob Group MeetingWednesday, January 23, 2008

Jan 23, 2008 Bob Y – Jacob Group Meeting –

1. Model updates since my last group meeting (Dec 2006)

2. Current met field availability for GEOS–Chem

3. GEOS–Chem benchmarking updates

4. GEOS–5 validation and outstanding issues

5. Adoption of Earth System Model Framework and porting of

GEOS–Chem to IBM BlueGene architecture

6. Stuff in the pipeline

Table of Contents


Model updates since my last

group meeting (Dec 2006)


Internal version v7–04–11 was the development version in Dec 2006• Improvements and additions to chemistry

– Ability to do chemistry up to the location of the actual tropopause (Philippe, Brendan) --DONE!– Added 2 tracers (SOG4, SOA4) to track the SOA prod from isoprene + OH reaction (Daven Henze) -- DONE!– HO2 uptake by aerosols is now turned off in the SMVGEAR mechanism (Bastien Sauvage) – DONE!

• Improvements and additions to emissions– Option to use GFED2 biomass emissions for gas phase + aerosol species -- DONE!– Option to use David Streets' regional emissions for China and SE Asia -- DONE!– Option to use EDGAR global NOx, CO, SO2 emissions -- DONE!– Option to use BRAVO NOx, CO, SO2 emissions over Northern Mexico -- DONE!– New module for IPCC future emissions scenarios (primarily for GCAP) -- DONE!– Near-land lightning NOx emissions module – DONE (NOW REPLACED IN v7-04-13!)

• Updated Hg simulation– More recent Hg simulation updates still need to be standardized – READY FOR INCLUSION!

• Technical updates– Support for Sun 4100 platform (with AMD/Opteron chipset) – DONE!– Removal of obsolete GEOS-1 and GEOS-STRAT met fields – DONE!– Removal of support for obsolete LINUX_IFC and LINUX_EFC compilers – DONE!– Various minor bug fixes & updates for the GCAP simulation – DONE!

Review: state of the model at last grp mtg


• v7–04–13 is latest benchmarked internal release

• TPCORE advection now works with GEOS–5 met!

– Yuxuan Wang & Dan Chen are very close to a working TPCORE for GEOS–5 nested grids

• v8–01–01 (public release w/ GEOS–5) is very close

– A 1-year benchmark was done

– A couple of things still need attention

Model updates since my last group meeting!


• GEOS–Chem v7–04–13 is latest internal release

– H2 / HD offline simulation (Jaegle group @ UW)

– Major bug fix in SMVGEAR! (May & Philippe)

v7–04–13 is latest benchmarked internal release


• Basic unit of SMVGEAR work = the “block” = 24 grid boxes

– Each processor gets 24 G–C grid boxes to work on simultaneously

– Abs & rel error tolerance checks are applied to the whole block

• Internal iterations

– SMVGEAR iterates over all of the grid boxes in the block until the absolute & relative error tolerance for the block as a whole is satisfied.

– If SMVGEAR does not converge to a solution, it reduces the timestep adaptively and then starts re-iterating over the block once again

• The problem

– Some G–C users reported that SMVGEAR was giving them extremely high Ox and extremely low CO in a particular grid box

– This caused the G–C simulation to die w/ an error

SMVGEAR Bug in more detail


• The cause

– It was possible to have a single grid box in the block be very negative, while the overall local & global error tolerance was satisfied

– SMVGEAR would then reset the negative value to “zero”

– The “zero” concentration value caused problems later on in places where it showed up in the denominator or in a logarithm.

• The solution (Thanks May and Philippe! )

– If any of the 24 grid boxes in the block are negative, then:

• Revert to the conc's at the end of the last successful internal iteration

• We reduce the internal iteration timestep and begin again

• Repeat until convergence

• If it still does not converge, then stop with an error

SMVGEAR Bug in more detail


• GEOS–Chem v7–04–13 is latest internal release

– H2 / HD offline simulation (Jaegle group @ UW)

– Major bug fix in SMVGEAR! (May & Philippe)

– Improved lightning NOx emissions algorithm (Lee Murray)

• Eliminate NOx emissions per path length

• Scale tropics to 260 moles N/flash (cf. Randall Martin)

• Eliminate top-down scaling

• New OTD/LIS local redistribution (cf. Murray et al 2007 in prep)

– Bug fix in variable tropopause

• Cap tropopause at 200 hPa poleward of 60S and 60N

v7–04–13 is latest benchmarked internal release


Getting TPCORE to work w/ GEOS–5 met

Pressure at grid edges:Pe(IJL) = A(L) + [ B(L) * Psurf(IJ) ]

Pressure at grid centers:Pc(IJL) = [ Pe(IJL) + Pe(IJL+1) ] / 2

Near the surface, the hybrid grid has a “sigma-like” terrain-following coordinate.

Near the model top, the hybrid grid has fixed-pressure levels. The pressure at each level is the same for all latitudes & longitudes.

Pros: Results in much smoother winds near the top of the model, better STE.


From the GEOS–5 file spec document (p.14)

– “In the GEOS–4 eta files, one could compute the pressure on the edges by using the “ak” and “bk” values and the surface pressure; once the edge pressures were known, they could be used to compute the average pressure in the layer.”

– “In GEOS–5, the full 3-dimensional pressure variables are explicitly provided at both layer centers (PLijl) and layer edges (PLEijl). As of this writing the pressures reported are on a hybrid-sigma coordinate, and could be obtained from the “ak-bk” relationship. But this may change in the future and so users should rely on the reported 3-dimensional pressures and not attempt to compute them from “ak” and “bk”.

Getting TPCORE to work w/ GEOS–5 met


Our first attempts to modify the TPCORE transport code for GEOS–5 were not very successful. This tagged Ox simulation indicates very little mixing even after 7 years.


As a result of a telecon between Harvard and GMAO on 10/30/07

– GMAO agreed to give us the “A’s” and “B’s” which define the GEOS-5 grid. In this way, we can use the existing TPCORE code without having to modify it. Everyone agreed that this was the fastest solution (particularly since Harvard needs to get a working GEOS-5 code ASAP for ARCTAS support).

– NASA/SIVO has a newer version of the TPCORE transport code that takes mass fluxes instead of winds. However, this version is fully ESMF-ized. To use it in the current GEOS-Chem version (aka “GEOS-Chem classic”) would require more time than we have before ARCTAS.

– However, the newer transport code will be useful as we create the new, ESMF-ized GEOS-Chem (for porting to MPI and eventually to the BlueGene).


TPCORE now works with GEOS–5 met fields!

GMAO provided us with the A’s and B’s that are used by TPCORE to compute the pressure at

altitude from the surface pressure.

Ox tends to steady state

after 10 years!

NOTE: This Ox spinup

was used to create the

initial conditions for

the 1-year benchmark simulations

with GEOS–5.


• GEOS–Chem v8–01–01 is very close

– This will be the first version that is compatible with the GEOS–5 met fields (hence the version # change).

• We will also update our web site for the public release

– New GEOS–Chem user manual

– Wiki will be publicly visible and editable

• It’s currently only internally visible at Harvard

• I will speak as to some of the ongoing GEOS–5 issues in a bit ..

v8–01–01 will be next public release


Current met field availability

for GEOS–Chem


GMAO Met Field Comparison Chart

Current met field availability for GEOS–Chem

2004 – onward + 30 yr reanalysis (MERRA)

1985 – present (ending in Feb 2007)

2000 – 20021996 – 19981985 – 1995 Temporal Coverage

2 x 2.54 x 5 + native

2 x 2.54 x 5

1 x 1 nested2 x 2.54 x 5

2 x 2.54 x 5

2 x 2.54 x 5

Regridded to

HDF4—EOSHDF4–EOSHDF4–EOSBinary BinaryNative File Format

0.5 x 0.6671 x 1.251 x 12 x 2.52 x 2.5Native grid

72 hybrid55 hybrid48 sigma26 sigma20 sigma# of levels

GEOS–5 GEOS–4 GEOS–3 GEOS– STRAT

GEOS–1

Past Pres, Fut.

O B S O L E T E !!

GEOS–3 and GEOS–4 are both still supported in GEOS–Chem. GEOS–4 is currently still heavily used in GEOS–Chem research. However, GMAO has now turned off the GEOS–4 data

stream. GEOS–5 is now the only dataset currently being produced by GMAO.

We also have GCAP met available for several years but these are not shown on the tables below.


Extra Slides


2007

2006

2005

GEO

S5 4

x5

GEO

S5 2

x2.

5

GEO

S5 0

.5 x

0.6

67 n

este

d C

hina

GEO

S4 4

x5

GEO

S5 0

.5 x

0.6

67 n

este

d N

A

GEO

S 4

2x25

NOTE: This plot is now on the GEOS–Chem web page under “Simulation Years”

Met

fiel

d da

te

GEOS–4 ends on 3/31/07. We have late-look

data thru 2006.



2007

2006

2005

GEO

S5 4

x5

GEO

S5 2

x2.

5

GEO

S5 C

hina

GEO

S4 4

x5

GEO

S5 N

A

GEO

S 4

2x25

2004

2003

2002G

EOS4

4x5

GEO

S4 2

x25

GEO

S3 4

x5

GEO

S3 2

x25

GEOS4 1 x 1.25NOTE: GEOS-5 data exists for 2004, we just haven’t

processed it yet

GEOS-3 ends 10/31/02

Met

fiel

d da

te



GEO

S5 4

x5

GEO

S5 2

x2.

5

GEO

S4 4

x5

GEO

S5 N

A

GEO

S4 4

x5

GEO

S3 4

x5

GEO

S3 2

x25

GEO

S4 4

x5

GEO

S4 2

x25

GEO

S3 4

x5

GEO

S3 2

x25

2002

2001

2000GEO

S3 1

x1 n

este

d N

A

GEO

S3 1

x1 n

este

d C

hina

GEOS-4 continues back to 1985GEOS–3 begins Jan 2000(some non-std data for 1998 exists)

Met

fiel

d da

te


GEOS‒5 has more levels in the PBLVertical levels up to 4 km – PBL

GEOS–3 GEOS–4 GEOS–5

GEOS–5 has much finer resolution in the PBL!


GEOS–5 also has more levels in the lower FT

GEOS–3 GEOS–4 GEOS–5

Vertical levels – up to 30 km


GEOS–Chem

benchmarking updates


Example: Cold Bay, Alaska CO (ppbv)

Even updates that seem “minor” at first glance can have a major impact on GEOS–Chem results!20 ppbv

cf. Rynda Hudman

Upshot: We have to keep track of what changes go into G–C.For changes that affect the full-chemistry simulation, we need to perform 1-year benchmark simulations.

We didn’t do such a good job of that in the recent past.

Note spread of different G-C versions w/r/t the observations!


GEOS–Chem Version History (1-yr benchmarks)

Slide from Rynda HudmanChanges are w/r/t the previous version


• A hard object lesson was learned:

– We cannot rush revisions into the code quickly and then “pray to God” that it will all work the first time thru …

• We now must be more diligent in benchmarking G–C

– Especially for versions that contain revisions which could have a major impact on the full-chemistry simulation

• We ask for your understanding

– “It’ll be ready when it’s ready …”

– We know that delays are sometimes frustrating

– Sometimes we encounter “open-ended” bugs or problems that are hard to solve quickly (cf. SMVGEAR bug fix in v7–04–13)

– However, It’s better to take more time and ensure the validity of the G–C version than to find out a major flaw with it later on

Updates for GEOS–Chem benchmarking


• 1-month benchmarks will be done for each internal & public version (GEOS–4 and GEOS–5)

• When do we do a new 1-yr benchmark simulation?

– For updates to emissions

• Anthro, biofuels, biomass, lightning, etc.

– For changes to chemical mechanism

• Reaction rates

• Photolysis

– For dynamical changes

• New met fields (e.g. GEOS–4 vs. GEOS–5)

• New convection and/or advection schemes



• IDL codes for creating plots from 1-month and 1-year benchmarks have been overhauled (L. Zhang, Bob Y.)

• 1-yr benchmarks now include the following plots(3 models vs. observations)

– Profiles of differences along 15S and 42N

– Difference maps at surface and 500 hPa

– Seasonal cycle

– Model ozone profiles vs. sondes

– Seasonal cycles of model ozone and CO vs. observations

– Model CO vs CMDL and ship track observations

– Ozone & CO vs. MOZAIC observations

– Ozone & CO vs. observations from various aircraft missions



• 1-month benchmarks now include the following plots(this version vs. previous version)

– Frequency distribution

– J-Value ratios at surface and 500 hPa

– Tracer ratios at surface and 500 hPa

– Tracer absolute difference at surface and 500 hPa

– Profiles of differences along 15S and 42N

– Tracer concentration maps

– Budget

– Emissions totals

• Benchmark plots will be accessible from our FTP & web site



GEOS–5 validation

and outstanding issues


• A 1-yr benchmark simulation was done with GEOS–Chem v8–01–01 driven with GEOS–5 met– A few areas still need attention:

• Convection– GEOS–5 cloud convection is weaker than GEOS–4?

• Lightning

– Less LNOx produced w/ GEOS–5

– Lack of lightning over certain areas (e.g Central Asia)

• Wet deposition (rainout & washout)

– Differences in precip fields between GEOS–4 and GEOS–5

GEOS–5 Outstanding Issues


Strong convection; wet season

v7-04-12-Run2 (geos4); v7-04-13 (geos4); v8-01-01 (geos5); data

Dry season at these stations

Note that the GEOS–5 Ox profiles are

very similar to the simulations

done with GEOS–4 up to 200 hPa, but

diverge higher up.

This can be an indication that

cloud convection in

GEOS–5 is weaker than in

GEOS–4.

200 hPa

Plot by Inna Megretskaia


GEOS–4 vs. GEOS–5 cloud mass fluxes

NOTE: original unit of GEOS–4 cloud mass flux is Pa/s,

we have converted to kg/m2/s to compare w/

GEOS–5.

No cloud mass flux below ~1 km in GEOS–5!

Alti

tude

(km

)

0.8 km


GEOS–4 puts lightning in Central Asia, but GEOS–5 doesn’t.

Figure from Lee MurrayGEOS–4 LNOx (CTH) GEOS–5 LNOx (CTH)

We are currently looking

into this!

This is most likely due to lower cloud-top heights in GEOS–5.

Global LNOx total w/ GEOS–5 met: ~ 5.4 Tg N/yrGlobal LNOx total w/ GEOS–4 met: ~ 6 Tg N/yr

0 1.2e–04 flash/min/km2



This plot shows the effect that the lower lightning NOx total in GEOS–5 has on the concentrations of Ox at midlatitudes.

Ozone seasonal cycle at 4 stations and 3 altitudes

Note: GEOS–5 (blue line) is lower than GEOS–4 (red, green lines) in summer!

Ox

(ppb

v)

Plot by Inna Megretskaia



Plot by Inna Megretskaia Ozone seasonal cycle at 4 stations and 4 altitudes

Ox

(ppb

v)

QQ: GEOS–Chem Precipitation Fields (in wet dep module)

NOTE: The GEOS–5 precipitation is much higher at about 1km than in GEOS–4!

Also note GEOS– 5 has more conv precip over tropics!

Hongyu and Bob Y. are looking at the wet deposition in G–C with Rn-Pb-Be

simulations. This is a work in progress.


Adoption of ESMF & MPI

and Porting of GEOS–Chem

to IBM BlueGene/L architecture


• What is ESMF?– It’s a software framework for earth science models that is being

implemented at NASA and elsewhere– http://www.esmf.ucar.edu

• What ESMF will do:– It will make it easier to connect “pieces” of the same model

together– It will make it easier to take “pieces” of one model and mix them

with “pieces” from other models.– It connects “pieces” of models together regardless of which

language the models were written in (Fortran, C, C++, etc.)– It will probably eventually reduce model development time

(however, some up-front development will be needed)

• What ESMF will not do:– It will NOT improve the quality of the science of a model

ESMF Overview Summary


Diagnostics

With ESMF, you can reduce a CTM to a collection of “pluggable” black box “Components” with standard inputs & outputs

In the future, GEOS–Chem will be a configuration of ESMF-compliant

components instead of a

standalone model

NOTE: All of the data that flows into a component goes

in through the “front door”, I.e.

through argument lists. No more global common

blocks!

ESMF library functions

pass data into & out of

components!


• Harvard/SEAS has recently purchased an IBM BlueGene/L

– Several research groups in SEAS are moving to this platform

• Earthquake fault & thrust modeling

• Protein folding

• Quantum chemistry

– Advantage: provides a high-quality computational platform to several research groups

– Advantage: users outside Harvard would also be able to take advantage of this platform

• We were asked by the Dean of IT to consider porting GEOS–Chem to the BlueGene

– Philippe & I have sat in on a couple of BlueGene workshops

Porting GEOS–Chem to BlueGene


• BlueGene/L is a radically new supercomputer architecture

– 4 of the Top 10 fastest computers are BlueGene’s (top500.org)

– LLNL & DOE use BG’s to “have a blast”, among other things

• Why BlueGene?

– Need lots of computing power to solve today’s complex problems in the interdisciplinary sciences

– Advantage: massively parallel machine

• Think big: you have the opportunity to submit a job using hundreds or thousands of cores! (core = processor)

• Each small task is assigned to its own core

• I/O can be done in parallel

– BG architecture minimizes power consumption

Porting GEOS–Chem to BlueGene


IBM BlueGene/L Architecture

K. Jordan, IBM

Take notice! Users log in to the “front end” (which is a more traditional

Unix environment) and submit jobs to the BG/L via the

LoadLeveler queuing software.



NOTE: One must break down arrays

into chunks that are less than 256 MB!

Each core has 256 MB of memory and does not see the

memory of the other cores.

Cannot use OpenMP, must use

MPI!

Harvard SEAS has a 2-rack

BG/L machine

K. Jordan, IBM



K. Jordan, IBM

Very complex design!


• Radical rewriting of GEOS–Chem will be necessary

– Cannot use much of existing G–C due to BG/L memory per node

• Existing G–C requires 2+ GB of shared memory

– Parallel file system implies file format will have to be switched from binary to netCDF or HDF

• Cannot read binary sequential files on BlueGene

– Because we will want to interchange codes w/ NASA, we will have to pass data around via calls to ESMF/MAPL libraries

• Implies familiarity with ESMF programming interface

• But all is not lost …

– We can start with existing routines from NASA/SIVO and/or GMI

– We can rely on the expertise of the folks at NASA

Challenges in porting GEOS–Chem to B/G


• Technical things

– ESMF and MAPL libraries must be installed on BG/L

– Learning curve! Bob & Philippe will have to learn many new things, including MPI programming and ESMF & MAPL API’s

– We must also communicate these new things to the G–C user base

– Existing ESMF / MPI etc. documentation is not very good

• Human Resources

– Perhaps up to ½ of Bob's or Philippe's time will be spent on porting

– More time will be spent on strictly software issues

– Significant up-front work may be required before we get results

Challenges in porting GEOS–Chem


• Testing concerns

– We cannot move to the IBM BlueGene/L in one jump

– We must port the model to a more traditional Unix environment first

– On the BlueGene/L, you cannot run the model on one processor. You have to use at least 32. This makes single vs. multi-processor testing impossible on the BlueGene/L.

• Scalability

– At this point we don’t know how scalable a new G–C code would be on the BlueGene

– If the code does not scale well on BlueGene then there is no point in porting there

– Better to develop the ESMF-ized version of G–C on a more traditional computational environment first. If there are scalability problems then they will more than likely show up at this stage.

Challenges in porting GEOS–Chem


Stuff in the pipeline


List of revisions, in no particular order

• Add glyoxal etc. rxns to chemistry mechanism (May Fu, Hong Kong Polytechnic U.)

• Updated Hg simulation (Noelle Selin, now @ MIT)

• Interface CASA model with mercury simulation (Nicole Smith-Downey, Harvard)

• Emissions scale factors (A. van Donkelaar, Dalhousie)

• GFED2 8-day biomass emissions (P, Kasibhatla, Duke) – now in testing

• Aerosol phase transition (J. Wang, U. Nebraska @ Lincoln)

• Updated ozone quantum yields for photolysis from JPL

• Updated EPA/NEI99 emissions w/ California transport fix (available for use now but not in std code)

• CAIR (EPA 2020) ship emissions, accounting for offshore limits (P. LeSager)

For ARCTAS

• Construction of near-real-time simulation (Bob Y.)

Things to be added into GEOS–Chem

Bob Yantosca Software Engineer

Documents

Transcript of Bob Yantosca Software Engineer