Verification Activity in Sub-seasonal Forecasting at JMA

1

Climate Prediction Division / JMAYuhei Takaya

WWRP/THORPEX/WCRP Sub-seasonal to Seasonal Implementation Planning Meeting

2-3 December 2011, Geneva Switzerland

Thanks to N. Adachi

1. Verification of real-time operational forecastMonitoring of the performance of real-time operational forecasts on a routine basis.

2. Verification of hindcastAssessing the performance of forecast systems using hindcasts (re-forecasts) for every implementation of systems.

Verification of Sub-seasonal Forecast

Climate Prediction Division, JMA2

Verification of Hindcast

3

• 5 members from 3 initial dates in all months during last ~ 30 years (currently 1979-2009)

Setups of Hindcast

Climate Prediction Division, JMA4

5 members

10th 20th The last day of the month

• Verification of JMA monthly EPS is based on the SVS-LRF.

• Scores include deterministic scores (i.e., RMSE, ACC)as well as probabilistic scores (i.e., ROC, BSS).

• Reference data are JRA-25 re-analysis, GPCPprecipitation.

Verification Scores

Climate Prediction Division, JMA5

ROC scores for 2-29 days mean precipitation (I.C.: Nov. 30 1979-2009)

0.8

0.5

0.6

• Our sub-seasonal/seasonal forecasts are provided together with skill assessment based on hindcasts.Tokyo Climate Center: http://ds.data.jma.go.jp/tcc/tcc/products/model/

Provision of Skill Assessment

Climate Prediction Division, JMA6 http://ds.data.jma.go.jp/tcc/tcc/products/model/map/1mE/map1/pztmap.php

The number of ensemble members (5 members) is not enough to get precise assessments of the performance.

Supplementary experiments with a larger number of ensemble members for limited initial dates.

This would be done in collaborative work of development of integrated EPS.

Current verification focus on forecast skill rather than representation of processes.

More process-oriented verification would be needed. Representation from a viewpoint of climate modeling.

(e.g., energy budget at the surface and TOA, MJO, etc… )

Actions Needed for Hindcast Verification

Climate Prediction Division, JMA7

Verification of Real-time Forecasts

8

Score Dependence on Periods of Climatology

9-0.2

0

0.2

0.4

0.6

0.8

1

-0.2 0 0.2 0.4 0.6 0.8 1

81-1

0 平

年値

79-04 平年値

3-4週目

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

81-1

0平

年値

79-04 平年値

Anomaly correlation of Z500 in NHaveraged over 28 days

0

0.2

0.4

0.6

0.8

1

0 0.2 0.4 0.6 0.8 1

81-1

0 平

年値

79-04 平年値

2週目

0.6

0.7

0.8

0.9

1

0.6 0.7 0.8 0.9 1

81-1

0 平

年値

79-04 平年値

1週目

1981

-201

0 cl

imat

olog

y

Anomaly correlation of GPH500 for 1979-2004, 1981-2010 climatology (operational forecasts during 2009/03/26-2011/04/21)

1979-1994 climatology

28 days mean Week-1

Week-3&4Week-2

Climate Prediction Division, JMA

Score Dependence on Periods of Climatology

10-0.2

0

0.2

0.4

0.6

0.8

1

-0.2 0 0.2 0.4 0.6 0.8 1

81-1

0 平

年値

79-04 平年値

3-4週目

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

81-1

0平

年値

79-04 平年値

Anomaly correlation of Z500 in NHaveraged over 28 days

0

0.2

0.4

0.6

0.8

1

0 0.2 0.4 0.6 0.8 1

81-1

0 平

年値

79-04 平年値

2週目

0.6

0.7

0.8

0.9

1

0.6 0.7 0.8 0.9 1

81-1

0 平

年値

79-04 平年値

1週目

1981

-201

0 cl

imat

olog

y

Anomaly correlation of GPH500 for 1979-2004, 1981-2010 climatology (operational forecasts dufing 2009/03/26-2011/04/21)

1979-1994 climatology

28 days mean Week-1

Week-3&4Week-2

Lesson :

ACC scores are affected by change of climatology periods.

Lesson :

ACC scores are affected by change of climatology periods.

Interannual Variability of Scores

Climate Prediction Division, JMA11

T106L40 (Top 0.4hPa)BGM(NH)/26-member

BGM(NH+TRO)land analysis

improved BGM(TRO)improved Cu conv. Scheme

improved radiation Scheme

SSM/I

improved Cu conv. scheme

Sc scheme

TL159L40 / 50-memberimproved radiation scheme

COBE-SST (for B.C.)

TL159L60 (Top 0.1hPa)improved Cu conv. schemeimproved gr. wave scheme

Interannual Variability of Scores

Climate Prediction Division, JMA12

T106L40 (Top 0.4hPa)BGM(NH)/26-member

BGM(NH+TRO)land analysis

improved BGM(TRO)improved Cu conv. Scheme

improved radiation Scheme

SSM/I

improved Cu conv. scheme

Sc scheme

TL159L40 / 50-memberimproved radiation scheme

COBE-SST (for B.C.)

TL159L60 (Top 0.1hPa)improved Cu conv. schemeimproved gr. wave scheme

Lessons from this exercise:

Forecast skill would be affected by the inherent predictability of the climate variability.

Sufficient length (samples) of forecast data is needed to verify the performance of sub-seasonal forecast systems. It is better to use both hindcasts and operational forecasts to assess the performance of forecast systems.

Lessons from this exercise:

Forecast skill would be affected by the inherent predictability of the climate variability.

Sufficient length (samples) of forecast data is needed to verify the performance of sub-seasonal forecast systems. It is better to use both hindcasts and operational forecasts to assess the performance of forecast systems.

• Ongoing work• Integrated verification environment for an experiment

system of integrated EPS (weekly-monthly) . • Re-assessment of operational forecast scores with the

consistent JRA-25 analysis. (Previously forecast skill has been routinely verified against the deterministic analysis at the time.)

• Future work…• More appropriate setups in order to verify EPS

performance. (5 members hindcasts are obviously NOT enough!)

• Verification of precipitation with in-situ observations.

Planned and Ongoing Activity in Verification

Climate Prediction Division, JMA13