Dubrovnik - EWGLAM/SRNWP 8-11/10/ 2007
COSMO strategy for Verification
Adriano RaspantiCOSMO WG5 Coordinator – “Verification and Case studies”
Head of Verification Section at Italian Met Service ([email protected])
with contributions by WG4 (Interpretation-PP) and WG5 people
Dubrovnik - EWGLAM/SRNWP 8-11/10/ 2007
MAIN PLANS (or projects)
• Advanced interpretation and verification of very
high resolution models (project by Pierre Eckert)
• Conditional Verification-VerSUS project
• COSI “The global Score” (COSMO Index)
COSMO strategy for Verification
Dubrovnik - EWGLAM/SRNWP 8-11/10/ 2007
Advanced interpretation and verification of very high resolution models
BackgroundBackground
The increase in resolution of the models will lead to a “proliferation” of grid points and also to an increase of noise in the forecasts.
The effects of the so-called “double penalty” also will increase for events not predicted exactly at the right place at the right time.
Ways to extract the most valuable information out of high density fields have to be found.
The connection with various fuzzy verification methods will be explored in this project.
Dubrovnik - EWGLAM/SRNWP 8-11/10/ 2007
Advanced interpretation and verification of very high resolution models
MAIN Goal of the projectMAIN Goal of the project
Data with a very high spatial (and temporal) variability like precipitation have to be treated with special care in order to avoid the double penalty syndrome.
Following methods have been identified in a first stage: Fuzzy verification, Contiguous Rain Area (CRA), Neighborhood methods, Fraction skill score, Intensity scale technique and similar
When the aggregation region is small, the scores are usually poor, but with an increasing averaging area the scores become very good
The goal is to find the smallest area in which the benefit of running a very high resolution model is present. This will be called the reliable scalereliable scale.
Not only the verification will be carried out at this “optimal” scale, but the products for forecasters and customers should also be designed at this scale (or scales).
Dubrovnik - EWGLAM/SRNWP 8-11/10/ 2007
Advanced interpretation and verification of very high resolution models
Other aspects of the projectOther aspects of the project
1. Application of “boosting” method for the detection of “special" weather parameters
• This method finds optimal choices for predictors which are proposed by the meteorologists. Good results with weather parameters not directly included in the model like fog or visibility are expected.
2. Use of very high resolution precipitation as input of the hydrological models. Studies and verification on the impact of this coupling
Dubrovnik - EWGLAM/SRNWP 8-11/10/ 2007
Which rain forecast would you rather use?
Mesoscale model (5 km) 21 Mar 2004
Sydney
Global model (100 km) 21 Mar 2004
Sydney
Observed 24h rain
RMS=13.0 RMS=4.6
Advanced interpretation and verification of very high resolution models
Some early results
Picture From B. Ebert
Dubrovnik - EWGLAM/SRNWP 8-11/10/ 2007
A Fuzzy Verification Toolbox
Fuzzy method Decision model for useful forecast
Upscaling (Zepeda-Arce et al. 2000; Weygandt et
al. 2004)Resembles obs when averaged to coarser scales
Anywhere in window (Damrath 2004), 50%
coveragePredicts event over minimum fraction of region
Fuzzy logic (Damrath 2004), Joint probability
(Ebert 2002)More correct than incorrect
Multi-event contingency table (Atger 2001) Predicts at least one event close to observed event
Intensity-scale (Casati et al. 2004) Lower error than random arrangement of obs
Fractions skill score (Roberts and Lean 2005) Similar frequency of forecast and observed events
Practically perfect hindcast (Brooks et al. 1998)Resembles forecast based on perfect knowledge of
observations
Pragmatic (Theis et al. 2005) Can distinguish events and non-events
CSRR (Germann and Zawadzki 2004) High probability of matching observed value
Area-related RMSE (Rezacova et al. 2005) Similar intensity distribution as observed
Advanced interpretation and verification of very high resolution models
Some early results
Dubrovnik - EWGLAM/SRNWP 8-11/10/ 2007
Perturbati
on
Type of forecast error Algorithm
PERFECTNo error – perfect
forecast!-
XSHIFT Horizontal translationHorizontal translation
(10 grid points)
BROWNIAN No small scale skill
Random exchange of
neighboring points
(Brownian motion)
LS_NOISEWrong large scale
forcing
Multiplication with a
disturbance factor
generated by large scale 2d
Gaussian kernels.
SMOOTHHigh horizontal diffusion
(or coarse scale model)
Moving window arithmetic
average
DRIZZLEOverestimation of low
intensity precipitation
Moving Window filter setting
each point below average
point to the mean value
Advanced interpretation and verification of very high resolution models
Some early results
Dubrovnik - EWGLAM/SRNWP 8-11/10/ 2007
Effect of „Leaking“ Scores
observation forecast
Problem: Some methods assume no skill at scales below window size!
pobs=0.5 pforecast=0.5
Assuming random ordering within window
yes no
yes 0.25 0.25
no 0.25 0.25
An example: Joint probability method
ForecastO
BS Not perfect!
Advanced interpretation and verification of very high resolution models
Some early results
Dubrovnik - EWGLAM/SRNWP 8-11/10/ 2007
Summary
-0,1
0,1
0,2
0,3
0,4
0,5
0,6
0,7
0
0,1
0,2
Up-
scaling
Any-
where in
Window
50%
cover-
age
Fuzzy
Logic
Joint
Prob.
Multi
event
cont. tab.
Intensity
Scale
Fraction
Skill
Score
Prag-
matic
Appr.
Practic.
Perf.
Hindcast
CSSR
Area
related
RMSE
Leaking ScoresXSHIFT
BROWNIAN SMOOTH
LS_NOISE DRIZZLE„Sensitivity Score“
STD
good
good
• Leaking scores show an overall poor performance
• “Intensity scale” and “Practically Perfect Hindcast” perform in general well, but …
• Many score have problem to detect large scale noise (LS_NOISE); “Upscaling” and “50% coverage” are beneficial in this respect
• Leaking scores show an overall poor performance
• “Intensity scale” and “Practically Perfect Hindcast” perform in general well, but …
• Many score have problem to detect large scale noise (LS_NOISE); “Upscaling” and “50% coverage” are beneficial in this respect
Advanced interpretation and verification of very high resolution models
Some early results
Dubrovnik - EWGLAM/SRNWP 8-11/10/ 2007
CV Project - VerSUS - Verification System Unified Survey
MAIN Goal of the Versus projectMAIN Goal of the Versus project
Development of a common and unified verification “package” including a Conditional Verification tool.
METHODMETHOD
The typical approach to CV could consist of the selection of one or several forecast products and one or several mask variables or conditions, which would be used to define thresholds for the product verification (e.g. verification of T2M only for grid points with zero cloud cover in model and observations). After the selection of the desired conditions, a classical verification tool for statistical indexes can be used.
The more flexible way to perform a selection of forecasts and observations is to use an “ad hoc database”, planned and designed for this purpose, where the mask or filter could be simply or complex SQL statements.
Dubrovnik - EWGLAM/SRNWP 8-11/10/ 2007
CV Project - VerSUS - Verification System Unified Survey
Main DB Modules
RDBMS features :• OBS e FCS data
• Data configuration to perform verification
• Verification results, Scorse and images
“daemon” process (Loader) to load data from different sources (e.g. MARS, districo DB, File system): BUFR format for obs and GRIB format for fcs
processes performing verifications through specific requests (Integration with “R” statistic package) and storing of resulting data
WEB GUI (server-client architecture)
Dubrovnik - EWGLAM/SRNWP 8-11/10/ 2007
OBS dataConfiguration data
for verification
Verification results(Scores and images)
Verification R
Web GUI
FCS dataLoader
MARS Districo DB
Usermanagement
Versus-DB
VerSUS - Architectural Design
Dubrovnik - EWGLAM/SRNWP 8-11/10/ 2007
StationForecast User/FE
Observation
Index
Dubrovnik - EWGLAM/SRNWP 8-11/10/ 2007
CV Project - VerSUS - Verification System Unified Survey
VERSUS DB has the following main areas:
• Users managing area
• Front-End area for Front-End setting up. Two main FE: the
loader FE for data ingestion, and scores FE for the execution
of verification indexes by means of “R” package library.
• Meteorological data area, for handling of observations
(surface and upper air) and forecasts data and their lookup
tables.
• Score criteria area that manages the definition of scores and
their applications.
• Output area that stores the scores and graphical output.
Dubrovnik - EWGLAM/SRNWP 8-11/10/ 2007
CV Project - VerSUS - Verification System Unified Survey
Main lookup tables:•Station: the list of punctual meteorological station that provides surface or upper air
observation data to VERSUS system. The attributes are name, nationality, latitude, longitude,
height, the WMO and/or ICAO code (if they exist) of the station. Moreover there is an unique
identifier of the station that VERSUS DB automatically assigns when a new station is defined
by means of Graphic User Interface (GUI)
•Obs_type: the list of observation types (templates) such as synop, temp, any other
observation data coded in BUFR format. That table is modified by means of a GUI
•Obs_parameter: the list of BUFR parameter codes, the meaning and input measurement.
This table is automatically updated whenever a new occurrence of BUFR parameter code
comes to the system.
•Model: the list of meteorological models verified VERSUS
•Grid: the list of grids that are defined in the section 3 of the grib.
•Fcs_parameter: data defined in the section 1 of the grib.
The lookup tables are managed by GUI or loader FE of the system, automatically, The lookup tables are managed by GUI or loader FE of the system, automatically,
whenever a new instance of them occurs.whenever a new instance of them occurs.
Dubrovnik - EWGLAM/SRNWP 8-11/10/ 2007
CV Project - VerSUS - Verification System Unified Survey
The selection criteria of the forecast and observation data is setting up
by
means of a GUI. The information that must be define are:
•Stratification (lat/lon, WMO name, morphological,….)
•The list of R-verification indexes to apply
•The observed parameter and its condition/filter, if any
•The forecast parameter (model, grid, parameter) and its condition/filter, if any
•The method of getting forecast data, such as nearest point, mean on a given
radius,…
•The start date and end date of the data or the frequency (monthly, weekly,
seasonal)
•Steps
•Pressure Levels (for upper air)
Dubrovnik - EWGLAM/SRNWP 8-11/10/ 2007
• Continuous parameters: Reduction of variance
RV = 1 – (RMSE prog / RMSE ref)2
where ref = persistence
• Categorical parameters: ETS
ETS = (R – „chance“) / (T –“chance“)R= number of obs events correctly forecastT = number of events which were either observed or forecasted
global score S like
COSMO-index COSI = S/S0 x100
Where S0 is the value of S the first year of computation
iii
ii
SSww
1S
COSI “The global Score” (COSMO Index)
Dubrovnik - EWGLAM/SRNWP 8-11/10/ 2007
Parameters
• total cloud amount [threshold: 0-2, 3-6, 7-8
• temperature [t2m, later: tmin, tmax]
• 10m- windvector
• precipitation [thresholds: 0.2, 2, 10 mm/6h]
COSI “The global Score” (COSMO Index)
Dubrovnik - EWGLAM/SRNWP 8-11/10/ 2007
Verification frequency
• All 3h− T2m, 10m-wind and cloudiness:
• @ 00, 03,…, 18, 21 UTC later on: tmin & tmx over 12h
• 6h-sums: precipitation
COSI “The global Score” (COSMO Index)
Dubrovnik - EWGLAM/SRNWP 8-11/10/ 2007
Which models ? Aggregation ?
• Start with COSMO-7
• But programming also for COSMO-2
• Temperature and windspeed: 1 gridpoint
• Precipitation: mean in a radius of 15km
• Cloudiness: mean in a radius of 30 km
COSI “The global Score” (COSMO Index)
Dubrovnik - EWGLAM/SRNWP 8-11/10/ 2007
COSI “The global Score” (COSMO Index)
List of stations:
• starting point: EWGLAM station list for verification
• selection based on availability of cloudiness each 3h per day
• plus „some more“ representative stations for COSMO-countries
THE_Score will be computed for each COSMO-country and different regions (W/N/E/S-Europe, Alps, smallest common region of all COSMO-xx, …)
Dubrovnik - EWGLAM/SRNWP 8-11/10/ 2007
COSMO strategy for VerificationConclusions
Advanced interpretation and verification of very high resolution modelsAdvanced interpretation and verification of very high resolution models
• Search for the “optimal scale” for verification and for representation of precipitation fields
• Fuzzy Verification score are a promising framework for verification of high resolution precipitation forecasts.• • Not all scores indicate a perfect forecast by perfect scores (Leaking scores).
• Choice of the scores: Upscaling, Intensity scale, Fraction skill score (?)
• End of the project expected for 2008
Dubrovnik - EWGLAM/SRNWP 8-11/10/ 2007
COSMO strategy for VerificationConclusions
VerSUS projectVerSUS project
• One tool for Verification and Conditional Verification
• DB powerful
• No “ad hoc” application to create verifications: only simple selections
• R-Integration (to add statistical Indexes only the “Verification Package” can be updated) – Community Knowledge
• User configurable using the GUI (Graphical User Interface)
• GUI WEB-based
• End of the project expected for 2008 (delivery of the package)
Dubrovnik - EWGLAM/SRNWP 8-11/10/ 2007
COSMO strategy for VerificationConclusions
COSI “The global Score” (COSMO Index)COSI “The global Score” (COSMO Index)
• Next future implementation
• Included in Common Verification Suite package (common fortran package for standard verifications, delivered in 2006 for COSMO community)
• Will be included in VERSUS package
• First results hopefully for COSMO GM of 2008
Top Related