First EUROBRISA Workshop17 March 2008The ENSEMBLES project Model uncertainty in climate forecasting...

First EUROBRISA Workshop 17 March 2008 The ENSEMBLES project

Model uncertainty in climate

forecasting from seasons to

decades: the ENSEMBLES project

Francisco J Doblas-ReyesECMWF, Reading, UK


● Description of the ENSEMBLES project.

● Hindcast production in ENSEMBLES: Streams 1 and 2.

● Diagnosis of the ENSEMBLES simulations and forecast quality assessment.

● Applications: the actual value of forecast systems.

● Data quality control and dissemination.

● Inclusion of NCEP CFS in a European multi-model.

Outline


● Integrated Project funded by the EC within the VIth FP, Sep 2004-Aug 2009, 69 partners.

● Main goal: development of an integrated probabilistic prediction system for time scales from seasons to decades, and beyond. Therefore, research on both seasonal-to-decadal (s2d) and anthropogenic climate change (ACC).

● S2d to be used to assess the reliability of forecast systems used for scenario runs, both from a climate modelling and user perspective, in a seamless framework.

● Forecast uncertainty: comparison of the merits of multi-model, perturbed parameters and stochastic physics approaches to deal with model error.

● Diversity of applications: health, crop yield, energy…

The ENSEMBLES project


● Model error is a major source of forecast uncertainty. Three approaches to deal with model error are being investigated in ENSEMBLES: multi-model, stochastic physics and perturbed parameters.

● The multi-model is built from ECMWF, Met Office, Météo-France operational activities and the DEMETER experience at IfM-Kiel, CERFACS and INGV.

● Perturbed parameter system stems from the decadal prediction system (DePreSys) created at the Met Office.

● Stochastic physics system uses the ECMWF stochastic backscatter system developed for medium-range forecasts.

● A major s2d experiment to determine the benefits of each approach is ongoing (~20,000 years of integrations).

Dealing with forecast uncertainty


● Three systems: multi-model (ECMWF, GloSea, DePreSys, Météo-France, IfM-Kiel, CERFACS, INGV), stochastic physics (ECMWF) and perturbed parameters (DePreSys).

● Hindcasts in two streams:o Stream 1: hindcast period 1991-2001, seasonal (7 months, May

and November start date), annual (14 months, November start date) and 2 decadal (1965 and 1994), 9 member ensembles, ERA40 initialization in most cases.

o Stream 2: As in Stream 1 but over 1960-2005, with 4 start dates for seasonal hindcasts, at least 1 for annual and at least one 3-member decadal hindcast every 5 years.

o Additional simulations: DePreSys_PP carries out 14-month hindcasts, a 10-year hindcast every year and a 30-year hindcast every 5 years + lots of sensitivity experiments from the other contributors.

S2d ENSEMBLES global experiment


Sea surface temperature RMSE (solid) and spread (dashed) averaged over the Niño3.4 region for the 1st November

(bottom row) start dates over the period 1991-2001. RMSE of simple persistence in dashed black.

All forecast systems beat persistence. Multi-model is the most skilful system, with highest reliability (RMSE~spread), in the first 6 months,

while the perturbed parameter system is as good for longer lead times.

Multi-model (5 models, 45 members)

ECMWF Stochastic Physics (9 members)

Perturbed Parameters (9 members)

Stream 1 annual hindcasts


Reliability diagrams for 1-month lead seasonal (JJA) precipitation above the upper tercile over the tropical band for the ENSEMBLES Stream 1 multi-model (left, 45 members), stochastic physics (centre, 9 members) and perturbed parameters (right, 9 members) hindcasts started in May over

the period 1991-2001 verified against GPCP. The Brier and ROC skill scores, along with 95% confidence intervals (in brackets) computed using

a bootstrap method, are shown on top of each panel.

Multi-model0.110 (0.060,0.160)0.415 (0.348,0.477)

Stochastic physics0.018 (-0.043,0.070)

0.347 (0.272,0.411)

Perturbed parameters0.016 (-0.054,0.083)0.339 (0.271,0.408)

Stream 1 seasonal hindcasts


Scores for southern South America precipitation from Stochastic Physics, Perturbed Parameters (both with 9-member ensembles) and Multi-model (5 models, 45 members). Sample values are shown with black dots along with 95% confidence intervals obtained using a bootstrap method (verified against GPCP over 1991-2001).

Anomaly correlation coefficient Ratio between spread and RMSEROCSS for anomalies

above the lower tercile




200

17

157

78

156

58

Ratio of spread to RMSE of the three different forecast systems for several regions, lead times (1-1, 2-4, 5-7 months), events (anomalies above/below

the upper/lower tercile), start dates (May and November) and variables (T2m, precipitation, Z500 and MSLP) computed over the period 1991-

2001. The inset numbers indicate the number of cases where a system is superior with 95% confidence.



115

25

203

9

33

106

ROC skill score of the three different forecast systems for several regions, lead times (1-1, 2-4, 5-7 months), events (anomalies above/below the

upper/lower tercile), start dates (May and November) and variables (T2m, precipitation, Z500 and MSLP) computed over the period 1991-2001. The inset numbers indicate the number of cases where a system is superior

with 95% confidence.


Stream 2

ROC area for Stream 2 JJA (start date 1st of May) 2-metre temperature above the upper tercile over the period 1960-2005 (left) and 1981-2005

(right, hindcast period for ECMWF System 3).

1960-2005 1981-2005


From Keenlyside et al. (2008)

10-year mean prediction of meridional overturning circulation and tropical Pacific SST from 3-member ensembles started every five years over 1955-2005.

Decadal predictions: IfM


Decadal predictions: IfM

From Keenlyside et al. (2008)

10-year mean prediction of global mean surface temperature from 3-member ensembles started every five years.


T2m difference with ERA40/OPS for 10-year ensemble integrations started on 1st Nov 1994 of different IFS cycles (HOPE ocean model).

Global land

NH land

Global ocean

Tropical oceans

Decadal predictions: ECMWF


First two years of predictions (anomalies wrt 1979-2001) from June 2005 (Jun 2005-May 2007) for uninitialized (NoAssim), initialized (DePreSys) and AR4 simulations. Comparison to NCEP analyses. Note the improved forecast for

initialized predictions, although not everywhere.

D. Smith & J. Murphy (Met Office)

Decadal forecasts: DePreSys


DePreSys: Stream 1 decadal predictions

Anomaly correlation coefficient for DePreSys_PP as a function of lead time (from 1-1 to 27-62). Each dot shows the result for a version of the model, the final dot of each set being for the ensemble mean. The coloured bars indicate the 95% confidence intervals obtained using a bootstrap method

with 1,000 samples.

Tropical precipitation Northern Hemisphere T850


DePreSys: Stream 1 decadal predictions

Scores for the DePreSys versus DePreSys_PP for several regions (tropics, Northern Hemisphere, Europe, North America), lead times (3-14, 15-26, 27-62 months), events (anomalies above/below the upper/lower tercile and above the median), start dates (May and November) and variables (T2m, Z500, T850, precipitation and MSLP) over the period 1991-2001.

The inset numbers give the cases when the value for a system is significantly above the corresponding value of the other.

BSSI RelSS

6

43

3

43

ROCSS

9

18


Thirty-year DePreSys forecasts started on March 2007 for uninitialized (NoAssim, 4 members) and initialized (DePreSys, 10 members) ensembles.

Surface temperature

THC intensity

D. Smith & J. Murphy (Met Office)

30-year climate forecasts: DePreSys


A user strategy: the end-to-end approach

• A broad range of forecast products might be offered, but a specific analysis of the user requirements is necessary.

• End-to-end is based on collaboration and continuous feedback.

• End users develop their models taking into account climate prediction limitations.

• Users should use objective records of performance.

• The final level of forecast quality that provides added value is defined by the application -> user-oriented verification. End users should assess the final value of the predictions.

• Forecast reliability becomes a major issue.


● The possibility of adaptation to climate change via a learning process taking place at the interannual time scale, when actions can be corrected, is an obvious way to achieve a high degree of integration (integration of time scales) between climate information providers and users.

● This implies:o That similar forecast information is provided in the climate

forecasting and climate-change contexts.

o Involvement of both climate scientists and end-users to consider the whole range of time scales. For instance, crop managers consider long-term climate change as a process that takes place on a yearly basis.

Adaptation using climate information


Seasonal prediction with dynamical models

N………… N-14321 Forecast

………… NN-14321 Downscaling

N………… N-14321Application

model

0

Forecast probability of PP Forecasts probability of malaria

0

non-linear transformation


Precipitation composites for the five years with the highest (top row) and lowest (bottom row) malaria incidence in Botswana for DJF DEMETER seasonal predictions (left) and CMAP (right).

Areas with epidemic malaria in

Africa

Quartiles define extreme events (outbreaks) for

malaria prediction

Dynamical predictions for malaria warning


Probabilistic predictions of standardised malaria incidence quartile categories in Botswana with five months lead time.

-- high malaria years

-- low malaria years

ROC Score Precipitation Incidence

Event DEMETER CMAP DEMETER

Very low 0.95 1.00 1.00

Very high 0.52 0.94 0.84

Very low malaria

Very high malaria

Available in March

Available in November

Gain in lead time for malaria warning


Botswana malaria index from Thomson et al. (2005) and incidence simulated by the process-based Liverpool malaria model (LMM) using ERA40.

-2.0

-1.0

0.0

1.0

2.0

1982 1984 1986 1988 1990 1992 1994 1996 1998 2000

Year

Ma

lari

a A

no

ma

ly

-0.6

-0.4

-0.2

0.0

0.2

0.4

0.6

Inc

ide

nc

e A

no

ma

ly

Malaria Index LMM incidence

There is a disagreement between both models for the year 2000: is it

due to the impact of extreme temperature or precipitation?

Interaction of variables (which might be triggered by climate change)

affects the user

From A. Jones (Univ. of Liverpool)

Gain in lead time for malaria warning

0.0

30.0

60.0

90.0

120.0

150.0

180.0

210.0

240.0

270.0

300.0

01/98 01/99 01/00 01/01

Rain

fall (

mm

per

mo

nth

)/M

on

thly

In

cid

en

ce

15.0

16.5

18.0

19.5

21.0

22.5

24.0

25.5

27.0

28.5

30.0

TM

ax

-5

(d

eg

C)

rain (mm per month) incidence per month tmax-5


• ENSEMBLES s2d intends to provide the scientific community with full access to the hindcasts and with automatic tools to perform different analysis. A public data dissemination system with innovative features has been developed.

• We are collaborating with some ENSEMBLES partners on:o Expanding the KNMI Climate Explorer for exploratory analysis.

o Developing a downscaling web portal at the University of Cantabria.

Public services


Diagnostics and downscaling

ClimateExplorer &

Downscalingportal

Hindcasts run/archived at ECMWF (access to member state users)

common dataatmosphere

MARS

common dataocean

additional data

ECFS

ECMWF firewall

Archiving and dissemination strategy

MARS clientOPeNDAP

server

ENSEMBLES public data server (5 Tb)

common dataatmosphere

common dataocean


• CFS:o Monthly 9-month long, 15-member ensembles hindcasts over the

period 1981-2005 with a coupled ocean-atmosphere.

o Atmospheric component initialized from NCEP analysis using lagged average method and ocean from an assimilated ocean state.

o T62L64 atmospheric resolution, 1°x1° (1/3° in the tropics) 40 levels ocean.

• EUROSIP:o ECMWF (11 members, 1981-2005 hindcast period), GloSea (15

members, 1987-2005) and Météo-France (5 members, 1993-2005, although now updated to 11 members and 1981-2005) integrated monthly following the EUROSIP strategy.

o Common period 1993-2005.

EUROSIP and NCEP CFS


Precipitation biases against GPCP for summer (JJA, May start date) over the period 1993-2005

ECMWF Met Office

Météo-France CFS

EUROSIP and NCEP CFS: biases


Top row: SST RMSE (solid) and spread (dashed) for the Niño3.4, Niño4 and Niño1.2 regions against Reynolds for the May and November start dates

over the period 1981-2005. Bottom row: SST ACC

Multi-model has a similar skill to System 3; CFS does not seem to add much skill, although individual forecasts are fairly different; CFS has more

variability than System 3

Persistence System 3 CFS Multi-model

Niño3.4 Niño4 Niño1.2

EUROSIP and NCEP CFS: tropical SSTs


Scatter plots of EUROSIP vs EUROSIP+CFS multi-model forecast quality over 1993-2005 in terms of reliability skill score (RelSS) and ROC skill

score (ROCSS). Symbols indicate results for 19 regions, 2 start dates (May and November), 3 variables (T2m, MSLP, PP), 4 lead times (1-3, 2-4, 3-5, 4-6) and 2 events (lower and upper terciles). The numbers in brackets

indicate differences significant with a 95% confidence using a two-sample inference test..

RelSS ROCSS

EUROSIP and NCEP CFS: Forecast quality

101

3

77

22


Summary

● Stream 1 and 2 ensemble simulations are available from ECMWF. In the seasonal/annual time scale, the multi-model seems to provide better results overall.

● Decadal forecasting has made significant progress, forecast systems have been developed and preliminary attempts show that useful climate information is available.

● Progress has been made to increase the public access to the data.

● The inclusion of NCEP CFS in the EUROSIP multi-model increases the reliability of the ensemble (mainly because of the increase in the ensemble size) and the accuracy.

First EUROBRISA Workshop17 March 2008The ENSEMBLES project Model uncertainty in climate forecasting...

Documents

Transcript of First EUROBRISA Workshop17 March 2008The ENSEMBLES project Model uncertainty in climate forecasting...