© Crown copyright 2005 Page 1

SRNWP – Revised Verification Proposal

Clive Wilson, COSMO Annual Meeting September 18-21, 2007

© Crown copyright 2005 Page 2

Original Draft Proposal (not submitted to EUMETNET)

2 aims Development of a common verification package Realization of an operational model intercomparison

Further aims Provide new methods (fuzzy etc) Allow non-GTS observation data Radar composites (esp. OPERA)

Responsible Member would: Write & maintain code of package Compute comparison scores, website & archive Find NMS(or ECMWF) to host non-GTS data hub Motivate NMSs to contribute non-GTS data to hub

© Crown copyright 2005 Page 3

Not submitted because:

Not fully agreedToo ambitious- underestimate time/effort needed to develop new code & package

No one indicated wish to be responsible member

Major centres and consortia already had most of proposed functionality in own packages

EUMETNET reluctant to agree proposed cost (1FTE scientist + travel +25% programme manager)

© Crown copyright 2005 Page 4

New (draft) proposal – staged aims

1. Inititiate a “realistic” intercomparison based on Exchange of forecasts from main models at 3-4 centres

(format – GRIB then interoperability to define) Met Office NAE - 12km Hirlam reference - 15km Aladin France - 10km COSMO-EU - 7km

Use existing packages Accept different station selection, QC (difficult to

mandate/change at op. centres) Verify common scores for same parameters over

common areas Compare and contrast results to reach “consensus” Extension of existing precip. verification done by Met

Office

© Crown copyright 2005 Page 5

Current Intercomparison of precipitation forecasts

© Crown copyright 2005 Page 6

External Met Office website- European Precipitation comparison

http://www.metoffice.gov.uk/research/nwp/numerical/precipitation/emip.html

user : viewprecip

pwd: rain4you

Models: Hirlam reference COSMO-EU (DWD lokall) Aladin (MeteoFrance) UM – North Atlantic European COSMO-7 (MeteoSwiss)

Verified against British Isles Nimrod radar composite

© Crown copyright 2005 Page 7

© Crown copyright 2005 Page 8

External Met Office website- European Precipitation comparison

24h accumulations thresholds : 0.125, 0.25, 0.5, 1.0, 2.0, 4.0, 8.0, 12.0, 16.0, 20.0,

24.0, 32.0 and 48.0 mmScores

frequency bias Equitable Threat Score (ETS) log-odds ratio Extreme Dependency Score

00 UTC run of each model since the beginning of January 2004Means, timeseries & ascii contingency tables

© Crown copyright 2005 Page 9

Some Example results

© Crown copyright 2005 Page 10

Latest results

© Crown copyright 2005 Page 11

New (draft) proposal – later staged aims

2. Add higher resolution forecasts to intercomparison

3. Methods/code for high resolution forecasts Collaborate on investigation of new methods Intercomparison studies for set of forecasts from

single model(s) (cf NCAR project with WRF) Provide code for new methods Enable access to radar composites (OPERA)

4. Non-GTS data & hub Much greater financial and staff resources

© Crown copyright 2005 Page 12

Why new draft may be acceptable

Practical & pragmaticStage 1 does not involve large cost or require extensive code changes

Stage 1 addresses primary concern (of EUMETNET directors) for meaningful verification of operational models

Stage 2 addresses new challenge of high resolution – still active research topic

Stage 3 will be necessary to evaluate and assess future operational high resolution models

© Crown copyright 2005 Page 13

Questions & Answers