A Regional Climate Model Evaluation System: Facilitating the Use of Contemporary Satellite and Other Observations for Evaluating Regional Climate Model Fidelity

D. E. Waliser1,2, J. Kim2, C. Mattmann1,3, C. Goodale1, A. Hart1, P. Zimdars1and P. Lean1

National Aeronautics and Space Administration

Jet Propulsion LaboratoryCalifornia Institute of TechnologyPasadena, California

www.nasa.gov

Copyright 2010. All rights reserved.

Figure 1. The role of model evaluation in the model development process and uncertainty estimations.

Regional Climate Model Evaluation System (RCMES):•Provide a fast, flexible, comprehensive system to allow easy comparison of climate models with observations.•Enable researchers to handle a large volume of data and reduce time taken for model evaluation studies from weeks to hours.•Help model developers with cutting-edge observations and diagnostics to evaluate and improve their models.•Help end-users understand the uncertainties in climate projections for the regions of interest.•Efficient: Fast access to reference data and toolkit•User Friendly: Intuitive and transferrable GUI•Flexible: Cloud-based architecture•Expandable:

• Easy to add new data/analysis tool• Scalable storage solution

Regional Climate Model Evaluation System (RCMES):•Provide a fast, flexible, comprehensive system to allow easy comparison of climate models with observations.•Enable researchers to handle a large volume of data and reduce time taken for model evaluation studies from weeks to hours.•Help model developers with cutting-edge observations and diagnostics to evaluate and improve their models.•Help end-users understand the uncertainties in climate projections for the regions of interest.•Efficient: Fast access to reference data and toolkit•User Friendly: Intuitive and transferrable GUI•Flexible: Cloud-based architecture•Expandable:

• Easy to add new data/analysis tool• Scalable storage solution

Future works:1.Add additional reference datasets (e.g., other reanalysis, satellite data, in-situ)2.Examine remote sensing data for evaluating fine-scale (<10km) regional climate data.3.Additional metrics calculations and visualizations4.Improve GUI5.Use the system to evaluate regional/global climate models associated with National Climate Assessment (NCA), NARCCAP, CMIP5 and CORDEX (Africa and Asia).ReferenceHart, A.F., C.E. Goodale, C.A. Mattmann, P. Zimdars, D. Crichton, P. Lean, J. Kim, and D.E. Waliser, 2011: A cloud-enabled regional climate evaluation system. SECLOUD’11, May 22, 2011, Waikiki, Honolulu, HI, USA.

Future works:1.Add additional reference datasets (e.g., other reanalysis, satellite data, in-situ)2.Examine remote sensing data for evaluating fine-scale (<10km) regional climate data.3.Additional metrics calculations and visualizations4.Improve GUI5.Use the system to evaluate regional/global climate models associated with National Climate Assessment (NCA), NARCCAP, CMIP5 and CORDEX (Africa and Asia).ReferenceHart, A.F., C.E. Goodale, C.A. Mattmann, P. Zimdars, D. Crichton, P. Lean, J. Kim, and D.E. Waliser, 2011: A cloud-enabled regional climate evaluation system. SECLOUD’11, May 22, 2011, Waikiki, Honolulu, HI, USA.

RCMES overview:•Large database (MySQL + Apache Hadoop):

• Multiple reference datasets from:• Satellite remote sensing

• TRMM (1998-2010)• AIRS (2002-2010)• MODIS Cloudiness

• Analysis• CPC precipitation, CRU precipitation, 2-m air temperatures

• Assimilation• SWR (SNODAS; JPL&U. Colorado)

• Reanalysis• ERA-Interim (e.g. U(p), V(p), q(p), T(p), SLP)

• Extractors:• Process data from various data formats into a common

database schema.• Library of statistical metrics:

• Python routines with plug-ins in other languages (Fortran, c, idl) to calculate and plot standard metrics of model performance. (e.g. Bias, RMS error, Anomaly Correlation, Probability Distribution Functions).

RCMES overview:•Large database (MySQL + Apache Hadoop):

• Multiple reference datasets from:• Satellite remote sensing

• TRMM (1998-2010)• AIRS (2002-2010)• MODIS Cloudiness

• Analysis• CPC precipitation, CRU precipitation, 2-m air temperatures

• Assimilation• SWR (SNODAS; JPL&U. Colorado)

• Reanalysis• ERA-Interim (e.g. U(p), V(p), q(p), T(p), SLP)

• Extractors:• Process data from various data formats into a common

database schema.• Library of statistical metrics:

• Python routines with plug-ins in other languages (Fortran, c, idl) to calculate and plot standard metrics of model performance. (e.g. Bias, RMS error, Anomaly Correlation, Probability Distribution Functions).

1Jet Propulsion Laboratory, California Institute of Technology; 2 JIFRESSE, UCLA; 3University of Southern California

Background: Why model evaluation?•Climate model projections play a crucial role in developing plans to mitigate and adapt to climate variations and change for sustainable developments.•Assessing model performance is an important step in linking climate simulation quality to projection uncertainty and then to climate change impacts assessments.

• Uncertainties propagate according to model hierarchy

• Bias correction is based on model evaluation

• Determination of the weights in multi-model ensemble

•Model evaluation is also a fundamental part of model development and improvement (Figure 1).

Background: Why model evaluation?•Climate model projections play a crucial role in developing plans to mitigate and adapt to climate variations and change for sustainable developments.•Assessing model performance is an important step in linking climate simulation quality to projection uncertainty and then to climate change impacts assessments.

• Uncertainties propagate according to model hierarchy

• Bias correction is based on model evaluation

• Determination of the weights in multi-model ensemble

•Model evaluation is also a fundamental part of model development and improvement (Figure 1).

RCMED(Regional Climate Model Evaluation Database)A large scalable database to store data in

a common format

RCMET(Regional Climate Model Evaluation Toolkit)A library of codes for extracting data

from RCMED and model and for calculating evaluation metrics

Raw Data:Various Formats,

Resolutions,Coverage

MetadataMetadata

Data TableData Table

Data TableData Table

Data TableData Table

Data TableData Table

Data TableData Table

Data TableData Table

Common Format,Native grid,

Efficient architecture

Common Format,Native grid,

Efficient architecture

MySQLExtractorExtractor

TRMMTRMM

MODISMODIS

AIRSAIRS

SWESWE

ETCETC

Soil moisture

Soil moisture

Extract OBS data

Extract OBS data

Extract RCM data

Extract RCM data

RCM dataRCM datauserchoice

RegridderPut the OBS & RCM data on the

same grid for comparison

RegridderPut the OBS & RCM data on the

same grid for comparison

Metrics CalculatorCalculate comparison metrics

Metrics CalculatorCalculate comparison metrics

VisualizerPlot the metrics

VisualizerPlot the metrics

URL

User’s own

codes for

ANAL and VIS.

User’s own

codes for

ANAL and VIS.

Data extractor(Fortran binary)

Data extractor(Fortran binary)

Sample Graphical User interface

Select model data

Select Variable

2-m temperaturePrecipitationOLR (TOA)Cloud fraction10m wind speed

Next >Next >

Next >Next >

TRMMAIRS level III griddedERA-InterimURD SNODAS

Select Reference Dataset

Next >Next >

Select data period

Next >Next >

Select Data Timestep

DailyMonthlySeasonalAnnual

Next >Next >

Select Spatial Grid

Reference DataModel

Next >Next >

Select Metrics

Mean biasRMSEPattern correlationPDF Similarity scoreCoeff. of Efficiency Next

>Next

>

Select Plots

MapTime series

Process >Process >

Evaluation of the Simulated Cold Season Hydrology in CaliforniaWRF; Oct 2008 – Mar 2009; NCEP Final Analysis forcing

Evaluation of the Simulated Cold Season Hydrology in CaliforniaWRF; Oct 2008 – Mar 2009; NCEP Final Analysis forcing

WRF T2 (K): 00UTC AIRS T2 (K): Ascending passes (1:30PM) Bias (K): WRF-AIRS

Seasonal-mean 2-m Air TemperatureSeasonal-mean 2-m Air Temperature

Season-total Precipitation (mm): Multiple Reference Data

Season-total Precipitation (mm): Multiple Reference Data Bias (mm): WRF-TRMM

WRFTRMM

CPC Bias (mm): WRF-CPC

Issues:• Remote sensing data:

• Satellite fly-over timing• Sensor footprints

• Multiple REF data:

• Differences between REF datasets

• Reference data intercomparison

• Observational uncertainty

Issues:• Remote sensing data:

• Satellite fly-over timing• Sensor footprints

• Multiple REF data:

• Differences between REF datasets

• Reference data intercomparison

• Observational uncertainty

Evaluation of the CORDEX-Africa Multi-Model EnsemblePreliminary 20-year runs; 1989 –2008

Evaluation of the CORDEX-Africa Multi-Model EnsemblePreliminary 20-year runs; 1989 –2008

ENS

Overland mean (mm/day)

RMSE (mm/day)

Spatial Variability of the Precipitation Climatology using

Taylor diagram

Spatial Variability of the Precipitation Climatology using

Taylor diagram

Precipitation Annual Cycle in 6 Regions using Portrait diagram

Precipitation Annual Cycle in 6 Regions using Portrait diagram

RMSERMSE CorrelationCorrelation

Intuitive presentation schema can facilitate intercomparison of multiple modelsIntuitive presentation schema can facilitate intercomparison of multiple models

For more information, please email duane.waliser@jpl.nasa.gov