Post on 18-Jan-2016
A.Montani; Performance of the COSMO-LEPS system during DOPCOSMO meeting - Cracow - 15-19 September 2008
Performance of the COSMO-LEPS system
during D-PHASE Operations Period
Andrea Montani,
Chiara Marsigli and Tiziana Paccagnella
ARPA-SIM
Hydrometeorological service of Emilia-Romagna, Italy
X General COSMO meetingCracow, 15-19 September 2008
A.Montani; Performance of the COSMO-LEPS system during DOPCOSMO meeting - Cracow - 15-19 September 2008
Outline
• Introduction• Methodology of COSMO-LEPS• Verification results (SYNOP on the GTS):
– performance from 2002 onwards– performance during DOP
• Future plans
A.Montani; Performance of the COSMO-LEPS system during DOPCOSMO meeting - Cracow - 15-19 September 2008
COSMO-LEPS (developed at ARPA-SIM)
• What is it?It is a Limited-area Ensemble Prediction System (LEPS),
based on COSMO-model and implemented within COSMO (COnsortium for Small-scale MOdelling, which includes Germany, Greece, Italy, Poland, Romania, Switzerland).
• Why?It was developed to combine the advantages of global-
model ensembles with the high-resolution details gained by the LAMs, so as to identify the possible occurrence of severe and localised weather events (heavy rainfall, strong winds, temperature anomalies, snowfall, …)
generation of COSMO-LEPS to improve the Late-Short (48hr) to Early-Medium (132hr) range forecast of
severe weather events.
A.Montani; Performance of the COSMO-LEPS system during DOPCOSMO meeting - Cracow - 15-19 September 2008
Dim 2
Initial conditions Dim 1 Dim 2
Possible evolution scenarios
Dim 1 Initial conditions
ensemble size reduction
Cluster members chosen as representative members (RMs)
LAM integrations driven byRMs
LAM scenario
LAM scenario
LAM scenario
COSMO-LEPS methodologyCOSMO-LEPS methodology
A.Montani; Performance of the COSMO-LEPS system during DOPCOSMO meeting - Cracow - 15-19 September 2008
COSMO-LEPS suite @ ECMWF during D-PHASE Operations Period (from 1 June to 30 November 2007)
d-1d-1 dd d+5d+5d+1d+1 d+2d+2 d+4d+4d+3d+3
older EPSolder EPS
younger EPSyounger EPS
clustering clustering periodperiod
0000
1212
Cluster Analysis and RM identificationCluster Analysis and RM identification
4 variables4 variables
Z U V QZ U V Q
3 levels3 levels
500 700 850 hPa500 700 850 hPa
2 2 time time stepssteps
Cluster Analysis and RM identificationCluster Analysis and RM identification
European European areaarea
Complete Complete LinkageLinkage
COSMO-LEPS
Integration Domain
1616 Representative Representative Members driving the Members driving the 1616
COSMO-model COSMO-model integrations (weighted integrations (weighted according to the cluster according to the cluster
populations)populations)
employing either employing either Tiedtke or Kain-Fristch Tiedtke or Kain-Fristch
convection scheme convection scheme (randomly choosen) (randomly choosen)
COSMO-LEPS
clustering area
• suite running as a ”time-critical application” managed by ARPA-SIM;
• Δx ~ 10 km; 40 ML;• COSM0 v3.20;• fc length: 132h;• Computer time (4.7 million
BU for 2007) provided by the COSMO partners which are ECMWF member states.
A.Montani; Performance of the COSMO-LEPS system during DOPCOSMO meeting - Cracow - 15-19 September 2008
COSMO-LEPS suite @ ECMWFafter D-PHASE Operations Period (since December 2007)
d-1d-1 dd d+5d+5d+1d+1 d+2d+2 d+4d+4d+3d+3
older EPSolder EPS
younger EPSyounger EPS
clustering clustering periodperiod
0000
1212
Cluster Analysis and RM identificationCluster Analysis and RM identification
4 variables4 variables
Z U V QZ U V Q
3 levels3 levels
500 700 850 hPa500 700 850 hPa
2 2 time time stepssteps
Cluster Analysis and RM identificationCluster Analysis and RM identification
European European areaarea
Complete Complete LinkageLinkage
COSMO-LEPS
Integration Domain
1616 Representative Representative Members driving the Members driving the 1616
COSMO-model COSMO-model integrations (weighted integrations (weighted according to the cluster according to the cluster
populations)populations)
employing either employing either Tiedtke or Kain-Fristch Tiedtke or Kain-Fristch
convection scheme convection scheme (randomly choosen) (randomly choosen)
++perturbations in perturbations in
turbulence schemeturbulence scheme
COSMO-LEPS
clustering area
• suite runs as a ”time-critical application” managed by ARPA-SIM;
• Δx ~ 10 km; 40 ML; fc+ 132h;• COSM0 v4.0 since Dec07
(with RK + multi-layer);• computer time (5.8 million
BU for 2008) provided by the COSMO partners which are ECMWF member states.
A.Montani; Performance of the COSMO-LEPS system during DOPCOSMO meeting - Cracow - 15-19 September 2008
Dissemination during DOP
probabilistic products (www.smr.arpa.emr.it/prodotti and www.d-phase.info )
“deterministic” products (individual 16 COSMO-LEPS runs) derived probability products (EM, ES) images and alerts
A.Montani; Performance of the COSMO-LEPS system during DOPCOSMO meeting - Cracow - 15-19 September 2008
– SYNOP on the GTS
Objective verification of COSMO-LEPS
Main features:
variable: 12h cumulated precip (18-06, 06-18
UTC);
period : from Dec 2002 to Aug 2008;
region: 43-50N, 2-18E (MAP D-PHASEPHASE
area);
method: nearest grid point; no-weighted fcst;
obs: synop reports (about 470
stations/day);
fcst ranges: 6-18h, 18-30h, …, 102-114h, 114-126h;
thresholds: 1, 5, 10, 15, 25, 50 mm/12h;
system: COSMO-LEPS;
scores: ROC area, RPSS, Outliers
both (69) monthly and (23) seasonal scores were computed
A.Montani; Performance of the COSMO-LEPS system during DOPCOSMO meeting - Cracow - 15-19 September 2008
Time series of ROC area Area under the curve in the HIT rate vs FAR diagram. Valuable forecast systems have ROC area values > 0.6.
ROC the positive
impact of increasing the ensemble size in 2004 is evident for all thresholds and for different forecast ranges.
poor performance of the system in Spring and Summer 2006 (both particularly dry), despite system upgrades.
good performance during DOP 2007.
limited impact of system upgrades in 2008
A.Montani; Performance of the COSMO-LEPS system during DOPCOSMO meeting - Cracow - 15-19 September 2008
Seasonal scores of ROC area Performance of the system assessed for the last 5 summers (JJA) and the last 5 autumns
(SON). Consider the “event” 10 mm/12h; valuable forecast systems have ROC area values > 0.6.
ROC Need to take
into account the different statistics for each season (JJA 2004 and SON 2006 less rainy than the others).
Good scores obtained during DOP 2007 for either seasons.
Good scores for JJA 2008, more evident for longer ranges.
A.Montani; Performance of the COSMO-LEPS system during DOPCOSMO meeting - Cracow - 15-19 September 2008
Time series of Ranked Probability Skill Score
A sort of BSS “cumulated” over all thresholds. BSS is written as 1-BS/BSref. Sample climate is the reference system. BS measures the mean squared difference between forecast and observation in probability space.
Useful forecast system, if RPSS > 0.
RPSS the improvement of
the system performance is detectable for all forecast ranges along the years;
the poor performance of the system in summer 2006 is confirmed;
RPSS always positive throughout 2007 and especially high during DOP.
Lesser skill in 2008 (so far).
A.Montani; Performance of the COSMO-LEPS system during DOPCOSMO meeting - Cracow - 15-19 September 2008
Outliers: time series + seasonal scores How many times the analysis is out of the forecast interval spanned by the ensemble
members. … the lower the better … Performance of the system assessed as time series and for the last 5 summers (JJA).
OUTL Evident seasonal
cycle (more outliers in winter), but overall reduction of outliers in the years up to 2007.
Reduction of outliers from one summer to the other, related to the increase of ensemble size (evident for the 10m to 16m increase).
Need to take into account the different statistics for each season.
Best results for the last 2 summers.
A.Montani; Performance of the COSMO-LEPS system during DOPCOSMO meeting - Cracow - 15-19 September 2008
Main results
• Since November 2002, COSMO-LEPS system has been running on a daily basis.
• Since December 2005, COSMO-LEPS is a “member-state time-critical application”.
• COSMO-LEPS products are routinely used in met-ops rooms across COSMO community as well as in research Projects (e.g. PREVIEW: see the poster).
• During D-PHASE Operations Period, COSMO-LEPS products were used both in the Visualisation Platform and to feed hydrological models.
increase in ROC area scores and reduction in outliers percentages; positive impact of increasing the population from 5 to 10 members (June
2004); some deficiency in the skill of the system can be identified after the system
upgrades occurred on February 2006 (from 10 to 16 members; from 32 to 40 model levels), BUT
scores are encouraging throughout DPHASE Operations Period; system upgrades of Dec07 (RK + multi-layer + new perturbations)
brought modest positive impact.
Time-series verification scores indicate the following trends:
A.Montani; Performance of the COSMO-LEPS system during DOPCOSMO meeting - Cracow - 15-19 September 2008
Thank you !
A.Montani; Performance of the COSMO-LEPS system during DOPCOSMO meeting - Cracow - 15-19 September 2008
Present methodology
A.Montani; Performance of the COSMO-LEPS system during DOPCOSMO meeting - Cracow - 15-19 September 2008
Operational set-up
Core products: 16 perturbed COSMO-model runs (ICs and 3-
hourly BCs from 16 EPS members) to generate, “via weights”, probabilistic output (start at 12UTC; t = 132h);
Additional products: 1 deterministic run (ICs and 3-hourly BCs from the
high-resolution deterministic ECMWF forecast) to “join” deterministic and probabilistic approaches (start at 12UTC; t = 132h);
1 “hindcast” run (ICs and 3-hourly BCs from ECMWF analyses) to “downscale” ECMWF information (start at 00UTC; t = 36h).
A.Montani; Performance of the COSMO-LEPS system during DOPCOSMO meeting - Cracow - 15-19 September 2008
Recent updatesChanges to the operational suite on 1 December 2007:Changes to the operational suite on 1 December 2007:• new model version (COSMO-LM 4.0);• new numerics (Runge-Kutta scheme); • use of multi-layer soil model;• new perturbations introduced relative to the maximal turbulent length scale (tur_len) and
to the length scale of thermal surface patterns (pat_len) to increase spread and maintain skill.
Old system and new system ran in parallel for 52 days (October and November 2007);
Larger T2m spread in new system for all forecast ranges;
Comparable skill (in terms of T2m root-mean-square error) of the two systems.
back to plans
A.Montani; Performance of the COSMO-LEPS system during DOPCOSMO meeting - Cracow - 15-19 September 2008
Time series of ROC area for COSMO-LEPS (2)
Area under the curve in the FAR vs HIT diagram. Valuable forecast systems have ROC area values > 0.6
ROC
the positive impact of increasing the ensemble size in 2004 is evident for all thresholds and for different fcst ranges.
poor performance of the system in Spring and Summer 2006, despite system upgrades.
Jun04: 5m 10m
Feb06: 10m 16m; 32ML 40 ML
fc step: 78-90h
A.Montani; Performance of the COSMO-LEPS system during DOPCOSMO meeting - Cracow - 15-19 September 2008
Time series of Brier Skill Score BSS is written as 1-BS/BSref. Sample climate is the reference system. Useful forecast systems
if BSS > 0. BS measures the mean squared difference between forecast and observation in probability
space. Equivalent to MSE for deterministic forecast.BSS
improvement of performance detectable for all thresholds along the years;
still problems with high thresholds, but good trend in 2007.
fc step: 30-42h
Jun04: 5m 10m
Feb06: 10m16m; 32ML 40 ML
A.Montani; Performance of the COSMO-LEPS system during DOPCOSMO meeting - Cracow - 15-19 September 2008
Semi-diurnal cycle in COSMO-LEPS scores
BSS score … the higher the better … Performance of the system assessed for 5 different Summers (JJA).
BSS Evident 12-hour cycle in
BSS scores (the same holds for RPSS, while less evident for ROC area scores).
Better performance of the system for “night-time” precipitation, that is for rainfall predicted between 18Z and 6Z (ranges 30-42h, 54-66h, …).
The amplitude of the cycle is somewhat reduced throughout the years and with increasing forecast range.
The bad performance in Summer 2006 is confirmed.
A.Montani; Performance of the COSMO-LEPS system during DOPCOSMO meeting - Cracow - 15-19 September 2008
Score dependence on the domain size (1)
Verification of COSMO-LEPS against synop reports over the MAP D-PHASE area (~ 470 stations; MAPDOM) and the full domain (~ 1500 stations; fulldom):
different statistics of the verification samples; up to now, performance of the system over the 2 domains assessed only for the
last 6 months (March-August 2007). difficult to draw general conclusions
A.Montani; Performance of the COSMO-LEPS system during DOPCOSMO meeting - Cracow - 15-19 September 2008
Score dependence on the domain size (2)
RPSS
RPSS score… the higher the better… (and positive).
ROC area… the higher the better… (and above 0.6). Smoother transitions from month to month in “fulldom” scores.
Slightly better performance of COSMO-LEPS over the MAPDOM, but the signal varies from month to month.
Higher predictability with orographic forcing?
Need to check individual regions and/or to stratify for type of stations.
OUTL
ROC
Outliers percentage … the lower the better.
A.Montani; Performance of the COSMO-LEPS system during DOPCOSMO meeting - Cracow - 15-19 September 2008
COSMO-LEPS
16-MEMBER EPS
51-MEMBER EPS
tp > 1mm/24h
tp > 5mm/24h
Average values (boxes 0.5 x 0.5)MAM06
As regards AVERAGE precipitation above these two threshols, the 3 systems have similar performance.
A.Montani; Performance of the COSMO-LEPS system during DOPCOSMO meeting - Cracow - 15-19 September 2008
Main results
• COSMO-LEPS system runs on a daily basis since November 2002 and it has become a “member-state time-critical application” at ECMWF ( partial involvement of ECMWF operators in the suite management).
• Time series scores identify positive trends in the performance of the system:• increase in ROC area scores and reduction in outliers percentages;• positive impact of increasing the population from 5 to 10 members (June
2004);• some deficiency in the skill of the system identified after the system
upgrades occurred on February 2006 (from 10 to 16 members; from 32 to 40 model levels + EPS upgrade!!!) need of further investigation.
• High-resolution verification shows better scores of COSMO-LEPS with respect to EPS in forecasting precipitation maxima within boxes.• No clear impact of the weighting procedure as regards precipitation, BUT COSMO-LEPS ensemble mean for 2m temperature has better skill (lower standard deviation to observations) with weighting the ensemble members according to cluster populations compared to the mean with unweighted members.
A.Montani; Performance of the COSMO-LEPS system during DOPCOSMO meeting - Cracow - 15-19 September 2008
Time series of ROC area Area under the curve in the HIT rate vs FAR diagram. Valuable forecast systems have ROC area values > 0.6.
ROC the positive
impact of increasing the ensemble size in 2004 is evident for all thresholds and for different forecast ranges.
poor performance of the system in Spring and Summer 2006 (both particularly dry), despite system upgrades.
good performance during DOP 2007.
Jun04: 5m 10m
Feb06: 10m 16m; 32ML 40 ML
fc step: 30-42h
DOP
fc step: 78-90h
Jun04: 5m 10m
Feb06: 10m 16m; 32ML 40 ML
DOP
A.Montani; Performance of the COSMO-LEPS system during DOPCOSMO meeting - Cracow - 15-19 September 2008
Seasonal scores of ROC area Performance of the system assessed for 5 different summers (JJA) and 5 different autumns
(SON). Consider the “event” 10 mm/12h; valuable forecast systems have ROC area values > 0.6.
ROC Need to take
into account the different statistics for each season (JJA 2003 and SON 2006 less rainy than the others).
Best scores obtained during DOP 2007 for either seasons.
A.Montani; Performance of the COSMO-LEPS system during DOPCOSMO meeting - Cracow - 15-19 September 2008
Time series of Ranked Probability Skill Score
A sort of BSS “cumulated” over all thresholds. BSS is written as 1-BS/BSref. Sample climate is the reference system. BS measures the mean squared difference between forecast and observation in probability space.
Useful forecast system, if RPSS > 0.
RPSS the improvement of
the system performance is detectable for all forecast ranges along the years;
the poor performance of the system in summer 2006 is confirmed;
RPSS always positive throughout 2007 and especially high during DOP.
Jun04: 5m 10m
Feb06: 10m16m; 32ML 40 ML
DOP
A.Montani; Performance of the COSMO-LEPS system during DOPCOSMO meeting - Cracow - 15-19 September 2008
Outliers: time series + seasonal scores How many times the analysis is out of the forecast interval spanned by the ensemble
members. … the lower the better … Performance of the system assessed as time series and for 5 different summers (JJA).
OUTL Evident seasonal
cycle (more outliers in winter), but overall reduction of outliers in the years.
Reduction of outliers from one summer to the other, related to the increase of ensemble size (more evident for the 5 to 10 increase).
Need to take into account the different statistics for each season (JJA 2003 less rainy than the others).
Best results for D-PHASE summer!
Jun04: 5m 10m Feb06: 10m 16m; 32ML 40 ML
DOP
A.Montani; Performance of the COSMO-LEPS system during DOPCOSMO meeting - Cracow - 15-19 September 2008
• Verification vs “DPHASE observations”:
– assess performance over different domains (North and South of the Alps),
– study individual case studies,
– consider basin-by-basin performance.
• “Think about” increasing horizontal resolution to 7 km.
• Calibrate COSMO-LEPS fcsts using reforecasts (F. Fundel ,
Meteoswiss).
Future plans
• Implement “seamless COSMO ensemble system” merging
COSMO-LEPS and COSMO-SREPS.
• Develop “hybrid” clustering technique (take boundary conditions
from a “grand-global” ensemble provided by mixing ECMWF-EPS and
UKMO-MOGREPS).