May 30, 2003 Tony Eckel, Eric Grimit, and Cliff Mass UW Atmospheric Sciences This research was...

May 30, 2003

Tony Eckel, Eric Grimit, and Cliff Mass

UW Atmospheric Sciences

This research was supported by the DoD Multidisciplinary University Research Initiative (MURI) program administered by the Office of Naval Research under Grant N00014-01-10745.

Overview

Review Ensemble Forecasting Theory and Introduce UW’s SREFs

Discuss Results of Model Deficiencies on SREF- Need for Bias Correction- Impact on ensemble spread- Impact on probabilistic forecasts skill

Conclusions

T

The true state of the atmosphere exists as a single point in phase space that we never know exactly.

A point in phase space completely describes an instantaneous state of the atmosphere.For a model, a point is the vector of values for all parameters (pres, temp, etc.) at all grid points at one time.

An analysis produced to run a model like the eta is in the neighborhood of truth. The complete error vector is unknown, but we have some idea of its structure and magnitude.

e

Chaos drives apart the forecast and true trajectories…predictability error growth.

EF can predicted the error magnitude and give a “probabilistic cloud” of forecasts.

12hforecast

36hforecast

24hforecast

48hforecast

T

48hverification

phasespace

e

a

uc

j

tg

n

2 1 0 1 2 34

2

0

2

4

6

7

-2.96564

Core ,i 2

Cent ,1 2

32.5 ,Core ,i 1 Cent ,1 1

M

T

T

Analysis Region

48h forecast Region

12hforecast

36hforecast

24hforecast

Diagram for: PME ACMEcore

or ACMEcore+

phasespace

Plug each IC into the MM5 to create an ensemble of mesoscale forecasts (cloud of future states encompassing truth).

1) Reveal uncertainty in forecast 2) Reduce error by averaging M 3) Yield probabilistic information

e

a

uc

j

tg

n

c

2 1 0 1 2 34

2

0

2

4

6

7

-2.96564

Core ,i 2

Cent ,1 2

32.5 ,Core ,i 1 Cent ,1 1

T

M

T

Analysis Region

48h forecast Regionphasespace

ACME’s Centroid

e

n

ac

u

tg

T

j

M

T

Analysis Region

48h Forecast Region

e

a

uc

j

tg

n

c

2 1 0 1 2 34

2

0

2

4

6

7

-2.96564

Core ,i 2

Cent ,1 2

32.5 ,Core ,i 1 Cent ,1 1

phasespace

ACME’s Mirrored Members

FP = 93%

Parameter Threshold(EX: precip > 0.5”)

FP = ORF = 72%

Fre

qu

en

cy

Initial State

Forecast Probability from an Ensemble

• EF provides an estimate (histogram) of truth’s Probability Density Function (red curve).

• In a large, well-tuned EF, Forecast Probability (FP) = Observed Relative Frequency (ORF)

24hr Forecast State 48hr Forecast State

Fre

qu

en

cy

• In practice, things get wacky from• Under-sampling of the PDF (too few ensemble members)• Poor representation of initial uncertainty• Model deficiencies

-- Model bias causes a shift in the estimated mean-- Sharing of model errors between EF members leads to reduced variance

• EF’s estimated PDF does not match truth’s PDF, and Fcst Prob Obs Rel Freq

UW’s Ensemble of Ensembles

# of EF Initial Forecast Forecast Name Members Type Conditions Model(s) Cycle Domain

ACME 17 SMMA 8 Ind. Analyses, “Standard” 00Z 36km, 12km1 Centroid, MM58 Mirrors

ACMEcore 8 SMMA Independent “Standard” 00Z 36km, 12km Analyses MM5

ACMEcore+ 8 PMMA “ “ 8 MM5 00Z 36km, 12km variations

PME 8 MMMA “ “ 8 “native” 00Z, 12Z 36km large-scale

Hom

egro

wn

Impo

rted

ACME: Analysis-Centroid Mirroring Ensemble

PME: Poor Man’s Ensemble MM5: PSU/NCAR Mesoscale Modeling System Version 5

SMMA: Single Model Multi-Analysis

PMMA: Perturbed-model Multi-Analysis

MMMA: Multi-model Multi-Analysis

Resolution (~ @ 45 N ) ObjectiveAbbreviation/Model/Source Type Computational Distributed Analysis

gfs, Global Forecast System, Spectral T254 / L64 1.0 / L14 SSINational Centers for Environmental Prediction ~55km ~80km 3D Var cmcg, Global Environmental Multi-scale (GEM), Spectral T199 / L28 1.25 / L11 3D VarCanadian Meteorological Centre ~70km ~100km eta, Eta limited-area mesoscale model, Finite 12km / L60 90km / L37 SSINational Centers for Environmental Prediction Diff. 3D Var gasp, Global AnalysiS and Prediction model, Spectral T239 / L29 1.0 / L11 3D VarAustralian Bureau of Meteorology ~60km ~80km

jma, Global Spectral Model (GSM), Spectral T106 / L21 1.25 / L13 OIJapan Meteorological Agency ~135km ~100km ngps, Navy Operational Global Atmos. Pred. System, Spectral T239 / L30 1.0 / L14 OIFleet Numerical Meteorological & Oceanographic Cntr. ~60km ~80km

tcwb, Global Forecast System, Spectral T79 / L18 1.0 / L11 OITaiwan Central Weather Bureau ~180km ~80km ukmo, Unified Model, Finite 5/65/9/L30 same / L12 3D VarUnited Kingdom Meteorological Office Diff. ~60km

“Native” Models/Analyses of the PME

Design of ACMEcore+

8 5 3 2 5 3 2 2 8 8 = 921,600

Total possible combinations:

vertical 36km 12km shallow SSTIC ID# Soil diffusion Cloud Microphysics Domain Domain cumulus Radiation Perturbation Land Use Table

MRF 5-Layer Y Simple Ice Kain-Fritsch Kain-Fritsch N cloud standard standard

avn plus01 MRF LSM Y Simple Ice Kain-Fritsch Kain-Fritsch Y RRTM SST_pert01 LANDUSE.TBL.plus1

cmcg plus02 MRF 5-Layer Y Reisner II (grpl), Skip4 Grell Grell N cloud SST_pert02 LANDUSE.TBL.plus2

eta plus03 Eta 5-Layer N Goddard Betts-Miller Grell Y RRTM SST_pert03 LANDUSE.TBL.plus3

gasp plus04 MRF LSM Y Shultz Betts-Miller Kain-Fritsch N RRTM SST_pert04 LANDUSE.TBL.plus4

jma plus05 Eta LSM N Reisner II (grpl), Skip4 Kain-Fritsch Kain-Fritsch Y cloud SST_pert05 LANDUSE.TBL.plus5

ngps plus06 Blackadar 5-Layer Y Shultz Grell Grell N RRTM SST_pert06 LANDUSE.TBL.plus6

tcwb plus07 Blackadar 5-Layer Y Goddard Betts-Miller Grell Y cloud SST_pert07 LANDUSE.TBL.plus7

ukmo plus08 Eta LSM N Reisner I (mx-phs) Kain-Fritsch Kain-Fritsch N cloud SST_pert08 LANDUSE.TBL.plus8

Perturbations to:

1) Moisture Availability

2) Albedo

3) Roughness Length

ACMEcore+

CumulusPBL

ACME

Total of 129, 48-h forecasts (Oct 31, 2002 – Mar 28, 2003) all initialized at 00z- Missing forecast case days are shaded

Parameters:- 36 km Domain: Mean Sea Level Pressure (MSLP), 500mb Geopotential Height (Z500)- 12 km Domain: Wind Speed @ 10m (WS10), Temperature at 2m (T2)

Research Dataset

36 km Domain (151127)

12 km Domain(101103)

Verification:

- 36 km Domain: centroid analysis (mean of 8 independent analyses, available at 12h increments)

- 12 km Domain: ruc20 analysis (NCEP 20 km mesoscale analysis, available at 3h increments)

November December January

February March

3 1 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 2 2 3 1 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 2 2 3 3 1 1 1 1 11 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1 2 3 4 5 6 7 8 9 0 1 2 3 4

1 1 1 1 1 2 2 2 2 2 2 2 2 2 2 3 3 1 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 2 1 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 2 25 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9

cmcg*

The ACME Process

STEP 1: Calculate best guess for truth (the centroid) by averaging all analyses.

STEP 2: Find error vector in model phase space between one analysis and the centroid by differencing all state variables over all grid points.

STEP 3: Make a new IC by mirroring that error about the centroid.

cmcgC cmcg*

Sea

Lev

el P

ress

ure

(mb)

~1000 km

1006

1004

1002

1000

998

996

994

cent

170°W 165°W 160°W 155°W 150°W 145°W 140°W 135°W

eta

ngps

tcwbgasp

avn

ukmo

cmcg

MSLP analysis south of the Aleutians at 00Z on Jan 16, 2003

tcwb centroid centroid + (centroid tcwb)

bias correction…

Overview

The two flavors of model deficiencies play a big role in SREF:

1) Systematic: Model bias is a significant fraction of forecast error and must be removed.

2) Stochastic: Random model errors significantly increase uncertainty and must be accounted for.

Bias Correction: A simple method gives good results

Model Error*: Impact on ensemble spread

Final Results: Impact of both on probabilistic forecasts skill

* bias-corrected

980

990

1000

1010

1020

1030

1040

1050

980 990 1000 1010 1020 1030 1040 1050

MSLP Forecast (mb)

MS

LP

An

aly

sis

(m

b)

Often difficult to completely remove bias within a model’s code Systematic but complex; involving numerics, parameterizations, resolution, etc.

Depend upon weather regime (time of day, surface characteristics, stability, moisture, etc.)

Cheaper and easier to remove bias through post-processing Sophisticated routines such as MOS require long training periods (years) The bulk of bias can be removed with the short term mean error

Need for Bias Removal

NGPS Forecast vs Analysis

Data Info

Single model grid point in eastern WA

Verification: centroid analysis

70 forecasts (Nov 25, 2002 – Feb 7, 2003)

Lead time = 24h

980

990

1000

1010

1020

1030

1040

1050

980 990 1000 1010 1020 1030 1040 1050

GASP Forecast vs Analysis

980

990

1000

1010

1020

1030

1040

1050

980 990 1000 1010 1020 1030 1040 1050

GFS Forecast vs Analysis

980

990

1000

1010

1020

1030

1040

1050

980 990 1000 1010 1020 1030 1040 1050

GFS-MM5 Forecast vs Analysis

TrainingPeriod

Bias-correctedForecast Period

TrainingPeriod


TrainingPeriod


Gridded Bias Removal

N

n nji

tjitji o

f

Nb

1 ,

,,,,

1 N number of forecast cases (14) fi,j,t forecast at grid point (i, j ) and lead time (t)oi,j verifying observation

For the current forecast cycle:

1) Calculate bias at every grid point and lead time using previous 2 weeks’ forecasts

2) Post-process current forecast to correct for bias:

tji

tjitji b

ff

,,

,,*,, fi,j,t bias-corrected forecast at grid point (i, j ) and lead time (t)*

November December January

February March

3 1 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 2 2 3 1 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 2 2 3 3 1 1 1 1 11 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1 2 3 4 5 6 7 8 9 0 1 2 3 4

1 1 1 1 1 2 2 2 2 2 2 2 2 2 2 3 3 1 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 2 1 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 2 25 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9

Spatial and Temporal Dependence of Bias

GFS-MM5 MSLP Bias at f24

Common Bias Forecast Error

> 1 too low < 1 too high

Spatial and Temporal Dependence of Bias

GFS-MM5 MSLP Bias at f36

Common Bias Forecast Error

> 1 too low < 1 too high

Bias Vs Corrected at f36

0.00

0.50

1.00

1.50

2.00

2.50

3.00

3.50

4.00

4.50

avn cmcg eta gasp jma ngps tcw b ukmo mean

MS

LP

RM

SE

(m

b2)

0%

5%

10%

15%

20%

25%

30%

35%

0 12 24 36 48Lead Time (hours)

% Im

pro

ve

me

nt avn

cmcg

eta

gasp

jma

ngps

tcwb

ukmo

mean

Bias Correction Results

0.00

0.50

1.00

1.50

2.00

2.50

3.00

3.50

4.00

4.50


bia

sed

bia

s-co

rrec

ted

PME


0.00

0.50

1.00

1.50

2.00

2.50

3.00

3.50

4.00

4.50


MS

LP

RM

SE

(m

b2)

0%

5%

10%

15%

20%

25%

30%

35%


% Im

pro

ve

me

nt avn

cmcg

eta

gasp

jma

ngps

tcwb

ukmo

mean

ACMEcore


0.00

0.50

1.00

1.50

2.00

2.50

3.00

3.50

4.00

4.50


bia

sed

bia

s-co

rrec

ted

ACMEcore+


0.00

0.50

1.00

1.50

2.00

2.50

3.00

3.50

4.00

4.50


bia

sed

bia

s-co

rrec

ted


0.00

0.50

1.00

1.50

2.00

2.50

3.00

3.50

4.00

4.50


MS

LP

RM

SE

(m

b2)

0%

5%

10%

15%

20%

25%

30%

35%


% Im

pro

ve

me

nt avn

cmcg

eta

gasp

jma

ngps

tcwb

ukmo

mean

Lead Time (hours)

Verification Rank

Pro

babi

lity

Verification Rank HistogramRecord of where verification fell (i.e., its rank) among the ordered ensemble members:

Flat Well calibrated EF (truth’s PDF matches EF PDF)

U’d Under-dispersive EF (truth “gets away” quite often)

Humped Over-dispersive EF

12

34

56

78

9

12

24

36

48

0.00

0.05

0.10

0.15

0.20

0.25

0.30

ACMEcore

Lead Time (hours)

Verification Rank

Pro

babi

lity





12

34

56

78

9

12

24

36

48

0.00

0.05

0.10

0.15

0.20

0.25

0.30

*ACMEcore

Model Error Impact on ensemble spread …

0.0

1.0

2.0

3.0

4.0

5.0

6.0

7.0

8.0

0 12 24 36 48

Lead Time

Var

ian

ce /

MS

E

*PME

MSE of *PME Mean

0.0

1.0

2.0

3.0

4.0

5.0

6.0

7.0

8.0

0 12 24 36 48

Lead Time

Var

ian

ce /

MS

E

*ACMEcore

*ACMEcore+

MSE of *ACMEcore Mean

Ensemble Dispersion

(MSLP)

Analysis Error

Error Growth due to Analysis ErrorE

nse

mb

le V

aria

nce

(m

b2 )

Error Growth due to Model Error

EF Mean’s MSE adjusted by n / n+1 to account for small sample size

MS

E o

f E

F M

EA

N

0.0

1.0

2.0

3.0

4.0

5.0

6.0

7.0

8.0

0 12 24 36 48

Var

ian

ce /

MS

E

0.0

1.0

2.0

3.0

4.0

5.0

6.0

7.0

8.0

0 12 24 36 48

Lead Time

Var

ian

ce /

MS

E

*PME

MSE of *PME Mean

*ACMEcore

*ACMEcore+

MSE of *ACMEcore Mean

12

34

56

78

9

12

24

36

48

0.00

0.05

0.10

0.15

0.20

*ACMEcore

Lead Time (hours)

Verification Rank

Pro

babi

lity





12

34

56

78

9

12

24

36

48

0.00

0.05

0.10

0.15

0.20

*ACMEcore+

Lead Time (hours)

Verification Rank

Pro

babi

lity





Lead Time (hours)

Verification Rank

Pro

babi

lity





12

34

56

78

9

12

24

36

48

0.00

0.05

0.10

0.15

0.20

PME

Impact of both on probabilistic forecasts skill

Explain probabilistic forecast verification

P(MSLP < 1001mb) by uniform ranks method, 36h lead time, Sub-domain AReliability Diagram Comparison

PME ACMEcore

Sample Climatology

Resolution Reliability Uncertainty Skill Score

Biased 0.1860 0.0010 0.2111 0.8714

Corrected 0.1859 0.00005 0.2111 0.8752


Biased 0.1860 0.0010 0.2111 0.8714

Corrected 0.1859 0.00005 0.2111 0.8752

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0

Forecast Probability

Ob

serv

ed R

elat

ive

Fre

qu

ency


0.1443 0.0013 0.1756 0.8138

0.1465 0.0002 0.1756 0.8330

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0


Ob

serv

ed R

elat

ive

Fre

qu

ency


0.1526 0.0008 0.1756 0.8641

0.1522 0.0002 0.1756 0.8655

36km Verification Sub-domain A

0.70

0.75

0.80

0.85

0.90

0.95

1.00

0 12 24 36 48

Lead Time (hours)

Sk

ill S

co

re

(re

lativ

e to

sa

mp

le c

lima

tolo

gy)

*PME

PME

*ACMEcore

ACMEcore

*ACMEcore+

ACMEcore+

~6hr improvement by bias correction

~11hr improvement by multi-model diversity and “global” error growth

Skill vs. Lead Time (Sub-domain A)

36km Verification Sub-domain B

Skill vs. Lead Time(all bias –corrected)

0.60

0.65

0.70

0.75

0.80

0.85

0.90

0.95

1.00

0 12 24 36 48

Lead Time (hours)

Sk

ill S

co

re

(re

lativ

e to

sa

mp

le c

lima

tolo

gy)

*PME

*ACMEcore

*ACMEcore+

0.60

0.65

0.70

0.75

0.80

0.85

0.90

0.95

1.00

0 12 24 36 48

Lead Time (hours)

36km Sub-domain A (2/3 ocean)P(MSLP < 1001mb)

Sample Climatology 23%

36km Sub-domain B (mostly land)P(MSLP < 1011mb)

Sample Climatology 20%

~11hr improvement by PME ~22hr improvement by PME

~3hr improvement by ACMEcore+

Conclusions

Caveats:- Consider non-optimal ICs and small EF size? (still fair comparison between PME and ACMEcore)- What about higher skill of PME members? (not so dramatic after bias correction)- Does higher resolution of MM5 make comparison unfair? (fitting to lower res. would decrease 2)

Why bother with ACMEcore?

PME is certainly more skilled at the synoptic level, but has little to no mesoscale info.

Should these conclusions hold true for mesoscale?

YES! Model deficiencies for surface variables (precip, winds, temperature) can be even stronger, so the effect on SREF may be even greater.Demonstrating that is now the focus of my research…

P(precip > 0.25” in 6hr)

An ensemble’s skill is dramatically improved by:

1) Correcting model bias

2) Accounting for model uncertainty

UW’s Ensemble of Ensembles

# of EF Initial Forecast Forecast Name Members Type Conditions Model(s) Cycle Domain

ACME 17 SMMA 8 Ind. Analyses, “Standard” 00Z 36km, 12km1 Centroid, MM58 Mirrors

ACMEcore 8 SMMA Independent “Standard” 00Z 36km, 12km Analyses MM5

ACMEcore+ 8 PMMA “ “ 8 MM5 00Z 36km, 12km variations

PME 8 MMMA “ “ 8 “native” 00Z, 12Z 36km large-scale

ACNE 9 hybrid? 8 Ind. Analyses, 9 MM5 00Z, 12Z 36km, 12kmMMMA 1 Centroid variationsPMMA

ACNE: Analysis-Centroid Nudged Ensemble

SMMA: Single Model Multi-Analysis

PMMA: Perturbed-model Multi-Analysis

MMMA: Multi-model Multi-Analysis

Pro

pose

d

Skill Score (SS ) Details

climclimperfect

clim 1BS

BS

BSBS

BSBSBSS

2

1

1

n

iii OBSFP

nBS SCSCSCORFN

nORFFPN

nBS i

M

iiii

M

ii

111 2

1

2

1

***

(reliability) (resolution) (uncertainty)

Brier Score

Brier Skill Score

yuncertaint

reliabiltyresolution

uncertainy00

uncertainyresolutionyreliabilit1

SS

n: number of data pairsFPi: forecast probability {0.0…1.0}ORFi: observation {0.0 = yes, 1.0 = no}

M : number of probability bins (normally 11)N : number of data pairs in the binFP*

i : binned forecast probability {0.0, 0.1,…1.0}ORF*

i : observation for the bin {0.0 = yes, 1.0 = no}SC : sample climatology (total occurrences / total forecasts)

Decomposed Brier Score (uses binned FP as in rel. diag.)

Skill Score

FP = 77.1%

For a certain threshold, say Ws 20kt, the FP is then simply the area under the PDF to the right (1 p value)

Ws = {16.5 21.1 27.3 29.3 33.4 37.4 40.2 47.8}

Wind Speed (kt)

Fre

quen

cy

Ideal Calculation of Forecast Probability (FP)Given a very large ensemble, a PDF could be found a grid point for any parameter (e.g., wind speed, Ws).

Unfortunately, we work with very small ensembles so we can’t make a good estimate of the PDF.Plus, we often do not even know what PDF shape to fit.

So we are forced to estimate FP by other means, for a set of Ws forecasts at a point such as:

Note: These are random draws from the PDF above

0 10 20 30 40 50 60 700

0.01

0.02

0.03

0.04

0.05

FP = 7/8 = 87.5%

FP = 7/9 + [ (21.1 – 20.0) / (21.1 – 16.5) ] * 1/9 = 80.4%

16.5 21.1 27.3 29.3 33.4 37.4 40.2 47.8

8/8 7/8 6/8 5/8 4/8 3/8 2/8 1/8 0/8

Democratic Voting FP

Uniform Ranks FP

9/9 8/9 7/9 6/9 5/9 4/9 3/9 2/9 1/9 0/9

“pushes” FP towards the extreme values, so high FP is normally over-forecast and low FP is normally under-forecast.

a continuous, more appropriate approximation.

20.0

FP = [ (1 – GCDF(50.0)) / (1 – GCDF(47.8)) ] * 1/9 = 8.5%

0 10 20 30 40 50 60 70 800

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1 a b

fraction = a / b

16.5 21.1 27.3 29.3 33.4 37.4 40.2 47.8

Uniform Ranks FP

9/9 8/9 7/9 6/9 5/9 4/9 3/9 2/9 1/9 0/9

FP When Threshold Falls in an Extreme Rank

0.0,-

50.0

Use the tail of a Gumbel PDF to approximate the fraction for the last rank.

0 10 20 30 40 50 60 70 800

0.01

0.02

0.03

0.04

0.05

x

xCDF expexp)(G

6ˆ s

ˆˆ x

FP = [ (1 – CDF(50.0)) / (1 – CDF(47.8)) ] * 0.17 = 13.0%

16.5 21.1 27.3 29.3 33.4 37.4 40.2 47.8

Weighted Ranks FP

1.0 0.83 0.72 0.62 0.54 0.45 0.36 0.27 0.17 0.0

Calibration by Weighted Ranks

0.0,-

50.0

Use the verification rank histogram from past cases to define non-uniform, “weighted ranks”.

The ranks to sum up and fraction of the rank where the threshold falls are found the same way as with uniform ranks, but now the probability within each rank is the chance that truth will occur there.

Sample Climatology

Skill Zone

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0


Ob

se

rve

d R

ela

tiv

e F

req

ue

nc

y

Uniform Ranks vs. Democratic Voting

Data Info

P(MSLP < 1002mb)

Verification: centroid analysis

70 forecasts (Nov 25, 2002 – Feb 7, 2003)

Applied 2-week, running bias correction

36km, Outer Domain

Lead time = 48h

UR

DV

References

Hamill, T. M. and S. J. Colucci, 1997: Verification of Eta-RSM short-range ensemble forecasts. Mon. Wea. Rev., 125, 1312–1327

Eckel, F. A., 1998: Calibrated probabilistic quantitative precipitation forecasts based on the MRF Ensemble. Masters Thesis, 133 pp

-----, and M. K. Walters, 1998: Calibrated probabilistic quantitative precipitation forecasts based

on the MRF Ensemble. Wea. and Fcst., 13, 1132–1147

May 30, 2003 Tony Eckel, Eric Grimit, and Cliff Mass UW Atmospheric Sciences This research was...

Documents

Transcript of May 30, 2003 Tony Eckel, Eric Grimit, and Cliff Mass UW Atmospheric Sciences This research was...