The Model Coupling Toolkit and the CCSM Next-Generation Coupler (NGC)

J. Walter Larson and Robert L. JacobMathematics and Computer Science

DivisionArgonne National Laboratory

Outline1. Prologue2. Introduction—The Accelerated Climate

Prediction Initiative (ACPI) Avant Garde project

3. Description of the Model Coupling Toolkit (and status)

4. Overall Design of the new Community Climate System Model (CCSM) Next-Generation Coupler (NGC)

Prologue

High-performance is crucial for earth system modeling applicationsA less obvious point—software quality is important, too: Well-tested and documented Modularity to reduce software complexity Extensibility

For simulation science, the software should embody the rigor and reproducibility of the scientific method

The CCSM

Community Climate System Model developed by the USA National Center for Atmospheric Research (NCAR)A coupled earth system model comprising the following component models:

Atmosphere Ocean Sea Ice Land-surface Flux Coupler

Long-term (2005) Goal: implement the CCSM in a high-performance software framework

From CCSM to a Framework…

1. Re-engineer component model code to maximize modularity

2. Render component models as components amenable to plug-and-play usage

3. Identify common utility functions that reside in multiple component models, and make them shared utilities

4. Absorb common flux coupler and shared utility routines part of the framework core

This is a big job…

ACPI Avant Garde

A Joint US Department of Energy / National Science Foundation project to apply modern software engineering practices to create a performance-portable version of the NCAR Community Climate System Model (CCSM)

Participants:DOE Laboratories: ANL, LANL, LBNL, LLNL,

and ORNLNCARNASA Data Assimilation OfficeWeb Site: http://www.mcs.anl.gov/acpi

ACPI Avant Garde

Project Term: 1 June, 2000 – 31 September, 2001

Current Objectives:1. Modular, high-performance atmosphere

GCM with a choice of three dynamical cores

2. Create a new, modular, extensible and parallel flux coupler

3. Optimization of the ocean GCM POP4. Implementation of parallel I/O in CCSMA grant application for DOE funding through

2006 to expand ACPI has been submitted

A Typical Coupled Model

AGCM

Temp, Rain, Radiation, Wind

Sea Ice

Temp, Albedo

OGCMTemp, Currents, Salinity

Land Surface

Temp, Soil Moisture

Coupled Modeling System

Our Objectives

Create a model coupling environment that is: Flexible Extensible Performance Portable—supports

message-passing, shared-memory, and hybrid parallelism

Highly configurable and easy-to-use

The Flux Coupler

The Coupler has two basic functions:1. Command and Control2. Data flow between component models

(including grid interpolation, time averaging and accumulation, flux physics calculations, and merging of state and flux data from multiple component models).

We need to support both functions, but need solutions that are highly extensible and configurable

The model coupling toolkit (mct) is designed to support function number two above.

mpeuProvides the following services:

F90 module-style access to MPIPortable/Flexible definition of typesSupport for multiprocessor stdout/stderrError handling / shutdownTiming/Load balance monitoring toolsSorting ToolsSupport for basic derived types on which low-level classes in the mct are built

Code used as scaffolding, but utilities might fit well in CCSM.

The model coupling toolkit provides a foundation for parallel coupled models, with general utilities for:

1. Data decomposition 2. Flexible, extensible, indexible field storage3. Time averaging and accumulation4. Field interpolation5. Intercomponent communications and parallel

data transferThe mct eases the construction of coupler

compuational cores and component-coupler interfaces

The model coupling toolkit

Software description:1. Highly Modular, Implemented in Fortran 902. Approximately 10,000 Lines of Code3. Functions have extensive argument and

error checking4. Modules and routines have prologues that

can be processeed using ProTeX to create LaTeX documentation

5. Model Coupling Toolkit Web Site: http://www.mcs.anl.gov/~larson/mct

The model coupling toolkit

MCT Assumptions (so far; subject to change)

1. Each component has its own MPI communicator2. Each component has a unique integer ID

(provided by MPH)3. Each component is on a distinct set of

processors4. Interpolation implemented as sparse-matrix

multiply5. Components can exchange only real and

integer data as groups of vectors6. The mct user supplies:

Consistent numbering schemes for grid points Interpolation Matrix Elements

Any number of components, any grid, any decomposition, any number of processors-per component is allowed

Features

1. Data decomposition and indexing:Data decomposition descriptors:GlobalMap (1-D)GlobalSegMap(multidimensional)

Local vector indexing—Navigator

Global to Local and Local to Global indexing

2. Flexible, extensible, indexible field storage—The AttrVect classInternal field representation in the mctForms the basis for other classes including Accumulator GeneralGrid SparseMatrix

3. Time Averaging and Accumulation—The Accumulator class:Support for: Time averaging Time summation

4. Grid Representation and Grid Interpolation:Grid representation—GeneralGrid Literal listing of grid point coordinates Any dimensionality of grid supported Gridpoint sorting supported Extensible storage allows for inclusion

of area and volume weights

4. Grid Representation and Grid Interpolation, Continued:Grid interpolation—SparseMatrix Storage of global and local row and column

indexing Storage of matrix elements Support for automatic decomposition of

matrices, and generation of vector decompositions based on matrix decompositions

5. Intercomponent communications and parallel data transfer:Registry of components and their process pools—MCTWorldExchange between components of data decomposition descriptorsParallel communications scheduler—the Router

Building the CCSM Next-Generation Coupler

The mct provides the “plumbing and wiring” support and compute core of the NGCOther modules needed for the NGC—

1. A diagnostics / flux physics library2. Component Model Interface routines

Component Model Interfaces

What’s needed (first cut)—1. Component model type identification2. Ability to pack/unpack model data into

AttrVect form3. Conversion of model data decomposition

descriptors into GlobalMap or GlobalSegMap form

4. Mechanisim to report the component model’s coordinate grid

5. A standardized scheme for identifying component model input/output field names

Component Model Interfaces

The path to a software framework for climate system modeling—

1. Points 1-5 listed on the previous slide2. Expansion of point 5 to include a

standard mechanisim for reporting component model metadata regarding compatibility with other component models

Future development of the mct:Support for on-line interpolation matrix element generation

Interface to SCRIP for horizontal Vertical interpolation support

Performance optimization and inclusion of OpenMPUpwards abstraction of data typesSupport for other data storage and data decomposition typesExtension to support dynamically load balanced component models (but fixed process pool sizes)Extension to support dynamically load balanced component models (dynamically varying process pool sizes)

Exciting, “within reach” applications of the mct:Inclusion of other components in CCSM (e.g., atmospheric chemistry, biogeochemistry)Use of globus-enabled MPI to construct coupled models with components residing on multiple platformsConstruction of a regional coupled modelConstruction of non-earth-science “supermodels” (e.g., reactive flow)Collaboration with the DOE Common Component Architecture (CCA) Forum