Antje Weisheimer Meteorological Training Course 27 April 2006 Antje Weisheimer Multi-model ensemble...

Antje Weisheimer Meteorological Training Course 27 April 2006

Antje Weisheimer

Multi-model ensemble predictions

on seasonal to decadal timescales


Introduction

A. Murphy (1993): What is a good forecast?

1. Consistency: correspondence between forecaster‘s best judgement and their forecasts

2. Quality: correspondence between forecasts and matching observations Multifaceted nature of forecast evaluation Measure-orineted and distribution-oriented

scores

3. Value: benefits realised by decision makers through the use of the forecasts


Structure

1. The multi-model concept

2. Examples: DEMETER – multi-model seasonal forecasts EUROSIP – Operational multi-model seasonal

forecasts at ECMWF ENSEMBLES – multi-model seasonal, interannual

and decadal forecasts Others

3. Summary and outlook


Sources of uncertainty in dynamical seasonal forecasting

initial conditions limited accuracy of observations ensemble forecasting technique

boundary conditionssoil moisture, sea ice, aerosols

model error model structure

complex representation of physical processes in models combination of different skilful, quasi-independent models into a multi-model ensemble

parameterisationsunresolved processes stochastic physical parameterisations, see lecture by Judith Berner

physical parameter valuesnot precisely known (eg., cloud related parameters) perturbed parameter approach (Murphy et al., 2004; Stainforth et al., 2005)

numerical representationresolution, truncation

unknown unknowns ?


climate system state space

verification

t=0

t=T

model 1

Model 2

The multi-model ensemble concept

model 2model 3


climate system state space

verification

t=0

t=T

The multi-model ensemble concept

multi-model ensemble


The multi-model ensemble concept: Basic scenarios

t=0t=T

A

t=0

t=T

B

t=0

t=TC

All single-model ensembles lie below / above the veri-fication

Multi-model is impro-ved because of error cancellation

One single-model ensemble provides the best forecast

compared to this, the multi-model can only be worse

however, compared to all other single-model ensembles, the multi-model is still improved

The verification lies beyond all single-model forecasts

multi-model is improved compared to poor models

however, multi-model is worse than good models

Is there ‘the best model’??Hagedorn et al. (2005)


The multi-model ensemble concept: case A

DEMETER one-months lead SST anomaly hindcasts (left) and cumulative distribution (right) for JJA 1988 at a single grid point in the tropical Pacific.

case A: error cancellation

multi-model single-modelverification

Hagedorn et al. (2005)

SST anomalies

model ranking

cumulative distributions


The multi-model ensemble concept: case B

The identification of ‘the best model’ depends critically on the aspect considered:

variable region season lead time choice of metric/skill score

There is no single best model!

rank

rank

rank

SST 1987

SST 1988 MSLP 1988


The multi-model ensemble concept: case C

SST anomalies

None of the single-model ensembles predicts the anomaly with Prob ≠ 0 the multi-model ensemble can never be better than the best single model,

but will always be better than the worse single-models Note: multi-model assigns a higher probability to negative anomalies than

most single-model ensembles Hagedorn et al. (2005)


Partner Atmosphere Ocean

ECMWF IFS HOPE

LODYC IFS OPA 8.3

CNRM ARPEGE OPA 8.1

CERFACS ARPEGE OPA 8.3

INGV ECHAM-4 OPA 8.2

MPI ECHAM-5 MPI-OM1

UKMO HadCM3 HadCM3

hindcast production period: 1958-2001

9 - member IC ensembles for each model

ERA-40 initial conditions SST and wind perturbations 4 start dates per year: 1st of

Feb, May, Aug, and Nov 6 month hindcasts

The DEMETER project

multi-model of 7 coupled general circulation models

http://www.ecmwf.int/research/demeter/


Feb 87 May 87 Aug 87 Nov 87 Feb 88 ...

7 models x 9 ensemble members

63-member multi-model ensemble

= 1 hindcast

The DEMETER project

Production for 1958-2001 = 44x4 = 176 hindcasts

multi-model of 7 coupled general circulation models


DEMETER: example of Nino3 SST hindcasts

Nino3 area

ECMWF CNRM UKMO MPI ERA40


rel. frequency that the verification (ERA-40) lies outside the multi-model ensemble bounding box, based on 6-hourly data

DEMETER: capturing the T2m 1989-1998 verification

rel. spread of the multi-model ensemble vs. climatology

under- over-dispersive

systematic errors

Weisheimer et al. (2005)


bounding box

inside outside

spread ens/clim

<1

>1




inside outside

<1

>1

1st month



2nd month

3rd month

capture rate over time

days

frac

tion

of g

rid p

oint

s (in

%)

start date:



rel. frequency that the verification lies out-side the ensemble bounding box

multi-model ens single-model ens



DEMETER: multi-model vs single-model

Relative ACC improvement of the multi-model compared to the single models for JJA from 1980-2001 (one month lead)

SST MSLP

Anomaly Correlation Coefficients (ACC)

multi-model baselinemodel ranking




multi-model baselinemodel ranking

SST MSLP

Relative improvement of the multi-model compared to the single models for JJA from 1980-2001 (one month lead) for different scores.

Hagedorn et al. (2005)Anomaly Correlation Coefficients (ACC), root mean square skill score (RMSSS),

Ranked Probability Skill Score (RPSS) and ROC Skill Score (ROCSS)

Tropics


DEMETER: Brier score of multi-model vs single-model

1959-2001

multi-model

sing

le-m

odel

Brier skillscore



DEMETER: Brier score of multi-model vs single-model

multi-model

sing

le-m

odel

Reliability skill score

Resolution skill score

sing

le-m

odel

multi-model


improved reliability of the multi-model predictions improved resolution of the multi-model predictions


BSSRel-ScRes-Sc

Reliability diagrams (T2m > 0)1-month lead, start date May, 1980 - 2001


0.0390.8990.141

0.0390.8990.140

0.0950.9260.169

-0.001 0.877 0.123

0.0650.9180.147

-0.064 0.838 0.099

0.0470.8930.153

0.2040.9900.213

multi-model



multi-model minus best single model

multi-model minus random chosen single model

RPSS, precipitation, 1-month lead, start

date November



Is the multi-model skill improvement due to

– increase in ensemble size?– using different sources of information?

An experiment with the ECMWF coupled model

and 54 ensemble members to assess

– impact of the ensemble size– impact of the number of models

DEMETER: impact of ensemble size


single-model [54 members] multi-model [54 members]

1-month lead, start date May, 1987 - 1999

DEMETER: impact of ensemble size

BSSRel-ScRes-Sc

Reliability diagrams (T2m > 0)1-month lead, start date May, 1987 - 1999

0.1700.9590.211

0.2220.9940.227



DEMETER: impact of number of models

realisations of different single-model ensembles with the same number of members

realisations of different multi-model combinations



DEMETER: prediction of tropical storms

GCMs nowadays are able to simulate tropical storms with a seasonal evolution and interannual variability consistent with observations over the western North Atlantic, eastern North Pacific and western North Pacific

frequency of simulated tropical storms is strongly correlated with interannual variability of observed large-scale circulation

operational monthly forecasts of tropical storm frequency at ECMWF (see lecture by Frédéric Vitart)

objective procedure for tropical storms detection tracks low vortices with a warm core above (Vitart et al., 2003)

quality of seasonal prediction of tropical storms may be improved by multi-model combination DEMETER (Vitart, 2006)


DEMETER: averaged number of tropical storms 1987-2001

80°S80°S

70°S 70°S

60°S60°S

50°S 50°S

40°S40°S

30°S 30°S

20°S20°S

10°S 10°S

0°0°

10°N 10°N

20°N20°N

30°N 30°N

40°N40°N

50°N 50°N

60°N60°N

70°N 70°N

80°N80°N

20°E

20°E 40°E

40°E 60°E

60°E 80°E

80°E 100°E

100°E 120°E

120°E 140°E

140°E 160°E

160°E 180°

180° 160°W

160°W 140°W

140°W 120°W

120°W 100°W

100°W 80°W

80°W 60°W

60°W 40°W

40°W 20°W

20°W

6 118.2 1728.1 27.67.3 6.4

14.1 8.2 13.9 8.9 9.1 5.9

ensemble size= 9Period : 1987-2001Tropical storm frequency per yearDEMETER: ECMWf assim

MODEL OBSERVATIONSMULTI-MODEL

Vitart (2006)


DEMETER: tropical storms interannual variability 1987-2001

multi-model forecast observations 2error

Vitart (2006)

1987 1988 1989 1990 1991 1992 1993 1994 1995 1996 1997 1998 1999 2000 2001

Year

23456789

1011121314151617181920212223242526

Trop

ical

Sto

rm N

umbe

r

MULTIMODEL: ECMWF LODYC UKMO CNRM CERFACS MPI SCNR

Forecast starting on 1st MayTropical Storm Frequency over the North Atlantic (JJASO)

RMS Error= 2.93( 3.65)Correlation=0.62( 0.99)

FORECAST Observations 2 Standard Deviations

North Atlantic r=0.62

1987 1988 1989 1990 1991 1992 1993 1994 1995 1996 1997 1998 1999 2000 2001

Year

123456789

101112131415161718192021222324252627282930

Trop

ical

Sto

rm N

umbe

r


Forecast starting on 1st MayTropical Storm Frequency over the Eastern North Pacific (JJASO)



Eastern North Pacific r=0.56

1987 1988 1989 1990 1991 1992 1993 1994 1995 1996 1997 1998 1999 2000 2001

Year

89

101112131415161718192021222324252627282930313233343536373839

Trop

ical

Sto

rm N

umbe

r


Forecast starting on 1st MayTropical Storm Frequency over the western North Pacific (JJASO)



Western North Pacific r=0.72

1987 1988 1989 1990 1991 1992 1993 1994 1995 1996 1997 1998 1999 2000 2001

Year

123456789

1011121314151617181920212223

Trop

ical

Sto

rm N

umbe

r


Forecast starting on 1st NovemberTropical Storm Frequency over the South Pacific (DJFMA)



South Pacific r=0.62


EUROSIP: European operational Seasonal-to-Interannual Predictions

Three coupled seasonal forecast systems:– ECMWF– Météo France– UK Met Office

All systems are running on ECMWF supercomputers

Hindcast periods– 1987-2001 for ECMWF and UK Met Office– 1993-2004 for Météo France

Development of multi-model products is ongoing


observed SST anomalies DJF 2005/2006

ECMWFEUROSIP: European operational seasonal multi-model predictions

-1

-1 -1

-0.5 -0.5

-0.5

0.5 0.5

0.5

0.5

0.5

0.5

0.5

0.50.5

0.5

0.5

1

1

1

-10 -10-9 -9-8-7-6-5 -3

-1.5 -1.5

-1.5

-1.5

-1.5

-1.5

1.5

1.5

1.51.5

1.5

1.5

3

DJF 05/06 SST anomlies (1958-2001)

-10

-2.5

-2

-1.5

-1

-0.5

0.5

1

1.5

2

2.5

10

Courtesy L.Ferranti

Météo France

UK MetOfficeensemble mean anomalies

forecasts started in Nov 2005


Prob (MSLP<lower tercile)

6

101.5

ECMWF Mean of 31 Uninitialised Analyses Valid: VT:00UTC 1 December 2005 to 00UTC 31 December 2005 Surface: mean sea level pressure/Surf: mean sea

-30

-25

-10

-6

-2

-1

1

2

6

10

25

30

Unweighted meanForecast start reference is 01/11/05Prob(MSLP < lower tercile)EUROSIP multi-model seasonal forecast

No significance test appliedDJF 2005/06

ECMWF/Met Office/Météo-France

75°S 75°S

60°S60°S

45°S 45°S

30°S30°S

15°S 15°S

0°0°

15°N 15°N

30°N30°N

45°N 45°N

60°N60°N

75°N 75°N

150°W

150°W 120°W

120°W 90°W

90°W 60°W

60°W 30°W

30°W 0°

0° 30°E

30°E 60°E

60°E 90°E

90°E 120°E

120°E 150°E

150°E

Produced from real-time forecast data

0..10% 10..20% 20..40% 40..50% 50..60% 60..70% 70..100%

-1.5

-1.5

-1.5

-1.5

1.5

1.5

1.5

1.5

1.5

1.51.5

1.5

1.5

1.5

1.51.5

1.5

1.5

1.5

3

3

3

3

3

3

333

4

4

ECMWF Mean of 31 Uninitialised Analyses Valid: VT:00UTC 1 December 2005 to 00UTC 31 December 2005 Surface: 2 metre temperature/Surf: 2 metre temp

-12-7-6-5-4-3-2-1-0.50.5123456712

Unweighted meanForecast start reference is 01/11/05Prob(2m temperature > upper tercile)EUROSIP multi-model seasonal forecast

No significance test appliedDJF 2005/06

ECMWF/Met Office/Météo-France

75°S 75°S

60°S60°S

45°S 45°S

30°S30°S

15°S 15°S

0°0°

15°N 15°N

30°N30°N

45°N 45°N

60°N60°N

75°N 75°N

150°W

150°W 120°W

120°W 90°W

90°W 60°W

60°W 30°W

30°W 0°

0° 30°E

30°E 60°E

60°E 90°E

90°E 120°E

120°E 150°E

150°E

Produced from real-time forecast data

0..10% 10..20% 20..40% 40..50% 50..60% 60..70% 70..100%

Prob (T2m > upper tercile)

EUROSIP: The European winter DJF 2005/2006Probabilistic multi-model forecasts

observed anomalies T2m MSLP


EUROSIP: The latest forecasts for JJA 2006

EUROSIP forecasts for JJA initialised on April 1st 2006

Chances for a warm and dry summer are…


stream 1 month 18-24

Three approaches to tackle model uncertainty: Multi-model: 7 coupled GCMs, each 9 IC ensemble members Perturbed physics: 2 coupled GCMs, each 9 IC ens. members Stochastic physics: 1 coupled GCM, 9 ensemble members

- hindcast production period: 1991-2001- seasonal runs (7 months): two start dates per year (May, Nov)- annual runs (14 months): at least one start date per year (Nov)- multi-annual/decadal runs (10 years): starting in 1965 and 1994 - model level data available for 3 of the multi-model GCMs

ENSEMBLES: seasonal, interannual and decadal predictions

EU funded Integrated Project 09/2004 - 08/2009http://ensembles-eu.metoffice.com/index.html

http://www.ecmwf.int/research/EU_projects/ENSEMBLES/index.html

public data dissemination


ENSEMBLES: seasonal, interannual and decadal predictions

SST anomalies in the Nino3 region

First decadal hindcast experiments using multi-model, stochastic physics and perturbed parameter ensembles

obs ECMWF ECMWF-CASBS(stoch. phys.)

GloSea(MetOffice)

DePreSyS(pert. phys.)


Others: IPCC AR4 multi-model climate change simulations

23 coupled state-of-the-art GCMs run with different emission scenarios

Weisheimer and Palmer (2005)

multi-model histograms

Probability of warm European summers in 2081-2100

1971-1990

A2 2081-2100

B1 2081-2100

A1B 2081-2100

A1B+A2+B1

95%

central data archive at PCMDI


Others: ensemble climate simulations with perturbed parameters

Quantifying Uncertainty in Model Predictions (QUMP) using a 53-member ensemble based on perturbed physical

parameters

pdf of climate sensitivity

climateprediction.net using a multi-thousand member grand ensemble generated by

distributed computing

Stainforth et al, 2005Murphy et al, 2004

temperature distribution


Summary

The quality of seasonal-to-decadal predictions may be improved by using combined forecasts produced by different models (multi-model ensemble forecasts).

Multi-model ensemble forecasting is a pragmatic and efficient method in filtering out model errors present in the individual ensemble forecasts.

Multi-model predictions yield, on average, more accurate predictions than either of the individual single-model ensembles (e.g., DEMETER).

The improvement is mainly due to more consistency and increased reliability.


Outlook

A. Murphy (1993): What is a good forecast?

1. Consistency: correspondence between forecaster‘s best judgement and their forecasts

2. Quality: correspondence between forecasts and matching observations Multifaceted nature of forecast evaluation Measure-orineted and distribution-oriented

scores

3. Value: benefits realised by decision makers through the use of the forecasts

Paco’s talk after coffee break!


References (I)

Doblas-Reyes, F.J., M. Déqué and J.-P. Piedelièvre, 2000: Multi-model spread and probabilistic seasonal forecasts in PROVOST. Q.J.R.Meteorol.Soc., 126, 2069-2088.

Doblas-Reyes, F.J., R. Hagedorn and T.N. Palmer, 2005: The rationale behind the success of multi-model ensembles in seasonal forecasting. Part II: Calibration and combination. Tellus, 57A, 234-252.

Hagedorn, R., F.J. Doblas-Reyes and T.N. Palmer, 2005: The rationale behind the success of multi-model ensembles in seasonal forecasting. Part I: Basic concept. Tellus, 57A, 219-233.

Joliffe, I.T. and D.B. Stephenson (Ed.), 2003: Forecast verification: A practitioner’s guide in atmospheric science. Wiley New York, 240pp.

Murphy, A.H., 1993: What is a good forecast? An essay on the nature of goodness in weather forecasting. Wea. Forecasting, 8, 281-293.

Murphy, J.M. et al, 2004: Quantification of modelling uncertainties in a large ensemble of climate change simulations. Nature, 430, 768-772.

Palmer, T.N. et al, 2004: Development of a European multi-model ensemble system for seasonal to inter-annual prediction (DEMETER). Bull. Am. Meteorol. Soc., 85, 853-872.

Stainforth et al, 2005: Uncertainty in predictions of the climate response to rising levels of greenhouse gases. Nature, 433, 403-406.


References (II)

Weisheimer, A., L.A. Smith and K. Judd, 2005: A new view of seasonal forecast skill: Bounding boxes from the DEMETER ensemble forecasts. Tellus, 57A, 265-279.

Weisheimer, A. and T.N. Palmer, 2005: Changing frequency of occurrence of extreme seasonal temperatures under global warming. Geophys. Res. Lett., 32, L20721, doi:10.1029/2005GL023365.

Vitart, F., D. Anderson and T. Stockdale, 2003: Seasonal forecasting of tropical cyclone landfall over Mozambique. J. Climate, 16, 3932-3945.

Vitart, F., 2006: Seasonal forecasting of tropical storm frequency using a multi-model ensemble. Q.J.R.Meteorol.Soc., 132, 647-666.

Special issue in Tellus (2005), Vol. 57A on DEMETER

Antje Weisheimer Meteorological Training Course 27 April 2006 Antje Weisheimer Multi-model ensemble...

Documents

Transcript of Antje Weisheimer Meteorological Training Course 27 April 2006 Antje Weisheimer Multi-model ensemble...