Evaluation of Selected Winter ’04/’05 Performance Results
-
Upload
tad-butler -
Category
Documents
-
view
28 -
download
2
description
Transcript of Evaluation of Selected Winter ’04/’05 Performance Results
![Page 1: Evaluation of Selected Winter ’04/’05 Performance Results](https://reader035.fdocuments.net/reader035/viewer/2022062718/56812b3a550346895d8f4ac5/html5/thumbnails/1.jpg)
Seth Linden and Jamie Wolff
NCAR/RAL
Evaluation of Selected Winter ’04/’05 Performance Results
![Page 2: Evaluation of Selected Winter ’04/’05 Performance Results](https://reader035.fdocuments.net/reader035/viewer/2022062718/56812b3a550346895d8f4ac5/html5/thumbnails/2.jpg)
Weather Forecast Verification
• Consensus (RWFS) forecast is compared to individual model components
• Air-temperature, dewpoint, wind-speed and cloud-cover forecasts– 18 UTC runs for the entire
season (1 November 2004 to 15 April 2005)
• Error (RMSE) calculated for:– Colorado Plains: 176 sites– Mountains: 119 sites
Blizzard of March 2003
![Page 3: Evaluation of Selected Winter ’04/’05 Performance Results](https://reader035.fdocuments.net/reader035/viewer/2022062718/56812b3a550346895d8f4ac5/html5/thumbnails/3.jpg)
Air temperature RMSE
Colorado Plains
RWFS
Colorado Mountains
![Page 4: Evaluation of Selected Winter ’04/’05 Performance Results](https://reader035.fdocuments.net/reader035/viewer/2022062718/56812b3a550346895d8f4ac5/html5/thumbnails/4.jpg)
Dewpoint RMSE
Colorado Plains
Forward Error Correction
Colorado Mountains
Due to 3-hour MOS data
![Page 5: Evaluation of Selected Winter ’04/’05 Performance Results](https://reader035.fdocuments.net/reader035/viewer/2022062718/56812b3a550346895d8f4ac5/html5/thumbnails/5.jpg)
Wind Speed RMSE
Colorado Plains
Colorado Mountains
![Page 6: Evaluation of Selected Winter ’04/’05 Performance Results](https://reader035.fdocuments.net/reader035/viewer/2022062718/56812b3a550346895d8f4ac5/html5/thumbnails/6.jpg)
Colorado PlainsCloud Cover
RMSE
Colorado Mountains
![Page 7: Evaluation of Selected Winter ’04/’05 Performance Results](https://reader035.fdocuments.net/reader035/viewer/2022062718/56812b3a550346895d8f4ac5/html5/thumbnails/7.jpg)
•The ensemble approach utilized by the RWFS does improve the predictions on average for all verifiable parameters
•No single model performs better for all parameters
•A blend of weather models will provide better results
Summary/Recommendations
![Page 8: Evaluation of Selected Winter ’04/’05 Performance Results](https://reader035.fdocuments.net/reader035/viewer/2022062718/56812b3a550346895d8f4ac5/html5/thumbnails/8.jpg)
Forecast Model Weights Used by the RWFS
• System automatically weights forecasts based on skill
• Distribution of weight values per lead time for air-temperature, dewpoint, and wind-speed– 18 UTC run on 3 May 2005
• Weights looked at for two sites:– Denver International Airport– I-70 at Genesse
Which models have the most skill?
![Page 9: Evaluation of Selected Winter ’04/’05 Performance Results](https://reader035.fdocuments.net/reader035/viewer/2022062718/56812b3a550346895d8f4ac5/html5/thumbnails/9.jpg)
Air Temperature Model Weights
Denver Int. Airport
ETA
GFSMOSMOSI-70 at Genesee
RUC
![Page 10: Evaluation of Selected Winter ’04/’05 Performance Results](https://reader035.fdocuments.net/reader035/viewer/2022062718/56812b3a550346895d8f4ac5/html5/thumbnails/10.jpg)
Dewpoint Model Weights
Denver Int. Airport
I-70 at Genesee
![Page 11: Evaluation of Selected Winter ’04/’05 Performance Results](https://reader035.fdocuments.net/reader035/viewer/2022062718/56812b3a550346895d8f4ac5/html5/thumbnails/11.jpg)
Denver Int. Airport Wind SpeedModel Weights
MM5I-70 at Genesee
WRF
![Page 12: Evaluation of Selected Winter ’04/’05 Performance Results](https://reader035.fdocuments.net/reader035/viewer/2022062718/56812b3a550346895d8f4ac5/html5/thumbnails/12.jpg)
Insolation Weights
0
100
200
300
400
500
600
700
800
12 0 12 0 12 0 12 0 12 0 12 0 12 0 12 0 12 0 12 0 12 0
Time (UTC)
W/m
2MSH_obEtaGFSMM5WRFeta_12gfs_12mm5_16wrf_16
1/29 1/30 1/31 2/1 2/3 2/4 2/5 2/62/2 2/7 2/8
•No one model consistently outperforms the others•MM5 and WRF forecast hourly instantaneous values, ETA forecasts 3-hour instantaneous values and GFS forecasts 3-hour averages
Clear Conditions
For MDSS static weights were applied:- 50/50 split between MM5 and WRF for the
0-23 hour forecast- All Eta for the 24-48 hour forecast
![Page 13: Evaluation of Selected Winter ’04/’05 Performance Results](https://reader035.fdocuments.net/reader035/viewer/2022062718/56812b3a550346895d8f4ac5/html5/thumbnails/13.jpg)
QPF Weights
Model GFS EtaMM5 2hr
MM5 3hr
MM5 4hr
Total MM5
WRF 2 hr
WRF 3hr
WRF 4hr
Total WRF RUC
MAV- MOS Total %
TOTAL MM5+WRF Contribution
QPF Weights (%) 9 11 15 12 10 37 17 14 12 43 0 0 100.00 80
•Due to a lack of quality precipitation observations static weights were applied•Weights fixed based on expert opinion•MM5 and WRF were given 80% of the total weight
![Page 14: Evaluation of Selected Winter ’04/’05 Performance Results](https://reader035.fdocuments.net/reader035/viewer/2022062718/56812b3a550346895d8f4ac5/html5/thumbnails/14.jpg)
•Weight distribution reflects that the corrected (dynamic MOS) NWS models (ETA, GFS, and RUC) had the most overall skill
•No one model dominates for all parameters•The limitation of the NWS models is their 3-hr temporal resolution
•WRF and MM5 were given the highest static weights for Insolation and QPF
Summary/Recommendations
![Page 15: Evaluation of Selected Winter ’04/’05 Performance Results](https://reader035.fdocuments.net/reader035/viewer/2022062718/56812b3a550346895d8f4ac5/html5/thumbnails/15.jpg)
Road Temp Observation Variance
• Tr variance across E-470 corridor– Shading by permanent structures or passing
clouds– Make/model/installation/age of temperature
sensors
![Page 16: Evaluation of Selected Winter ’04/’05 Performance Results](https://reader035.fdocuments.net/reader035/viewer/2022062718/56812b3a550346895d8f4ac5/html5/thumbnails/16.jpg)
E-470 Road/Bridge SitesColorado Blvd Platte Valley
(road and bridge)
6th Ave Pkwy
Plaza ASmokey Hill Rd
(road and bridge)
![Page 17: Evaluation of Selected Winter ’04/’05 Performance Results](https://reader035.fdocuments.net/reader035/viewer/2022062718/56812b3a550346895d8f4ac5/html5/thumbnails/17.jpg)
SCT BKN OVC
LOCAL TIME (19 = noon, 07 = midnight)
27 Nov 2004 28 Nov 2004
![Page 18: Evaluation of Selected Winter ’04/’05 Performance Results](https://reader035.fdocuments.net/reader035/viewer/2022062718/56812b3a550346895d8f4ac5/html5/thumbnails/18.jpg)
OVC CLRBKN SCT
LOCAL TIME (19 = noon, 07 = midnight)
29 Nov 2004 30 Nov 2004
![Page 19: Evaluation of Selected Winter ’04/’05 Performance Results](https://reader035.fdocuments.net/reader035/viewer/2022062718/56812b3a550346895d8f4ac5/html5/thumbnails/19.jpg)
Summary/Recommendations
• Large variations in observed road and bridge temperatures– Over relatively small area
(10s of miles)
• Makes prediction and verification of pavement temperatures very challenging– Difficult to establish ground
truth
![Page 20: Evaluation of Selected Winter ’04/’05 Performance Results](https://reader035.fdocuments.net/reader035/viewer/2022062718/56812b3a550346895d8f4ac5/html5/thumbnails/20.jpg)
Road/Bridge Forecast Verification
• Road and bridge temperature forecasts– Using recommended
treatments from MDSS
• Error (MAE) and bias calculated for:– For each lead time (0-48hrs)
18 UTC runs
– E-470: 6 roads/2 bridge (1 Nov 2004 – 15 Apr 2005)
– Mountains: 5 roads (1 Feb 2004 – 15 Apr 2005)
East bound lane of I-70 at the summit of Vail Pass
![Page 21: Evaluation of Selected Winter ’04/’05 Performance Results](https://reader035.fdocuments.net/reader035/viewer/2022062718/56812b3a550346895d8f4ac5/html5/thumbnails/21.jpg)
Consistent low bias
Lead Time (0 = 18 UTC = noon, 18 = 12 UTC = 6am)
Peak insolation
Morning hours
E-470 road sites
Perfect forecast
![Page 22: Evaluation of Selected Winter ’04/’05 Performance Results](https://reader035.fdocuments.net/reader035/viewer/2022062718/56812b3a550346895d8f4ac5/html5/thumbnails/22.jpg)
Lead Time (0 = 18 UTC ~ noon, 18 = 12 UTC ~ 6am)
Shadowing?
evening
morning
E-470 bridge sites
![Page 23: Evaluation of Selected Winter ’04/’05 Performance Results](https://reader035.fdocuments.net/reader035/viewer/2022062718/56812b3a550346895d8f4ac5/html5/thumbnails/23.jpg)
Lead Time (0 = 18 UTC = noon, 18 = 12 UTC = 6am)
evening
morning
CDOT mountain road sites
![Page 24: Evaluation of Selected Winter ’04/’05 Performance Results](https://reader035.fdocuments.net/reader035/viewer/2022062718/56812b3a550346895d8f4ac5/html5/thumbnails/24.jpg)
Summary/Recommendations
• Larger Tr differences during times of high solar insolation likely due to several factors:– Errors in measuring pavement skin temp– Mountain shading during low sun angle– Limitations in insolation prediction in models– Limitations in pavement heat balance model
• Simplified assumptions about pavement characteristics
• Tb analysis compromised by:– Sensors shadowed by bridge rail– Bias results suggest tuning may be beneficial
• Overall Issue:– Actual/Recommended treatments not the same
![Page 25: Evaluation of Selected Winter ’04/’05 Performance Results](https://reader035.fdocuments.net/reader035/viewer/2022062718/56812b3a550346895d8f4ac5/html5/thumbnails/25.jpg)
Case Study Analysis• 183 day demonstration
– 16 winter weather days• 10 light snow• 5 moderate snow• 1 heavy snow
![Page 26: Evaluation of Selected Winter ’04/’05 Performance Results](https://reader035.fdocuments.net/reader035/viewer/2022062718/56812b3a550346895d8f4ac5/html5/thumbnails/26.jpg)
November 27-29, 2004
• First significant snow storm of the season– 5-8” in the Denver area
• Large variations in parameter predictions– Forecast vs. observations
• Denver International Airport• Ta, Td, Wspd, Cloud Cover and Precipitation
• 12 UTC 28th examined – Captured the start time of event
![Page 27: Evaluation of Selected Winter ’04/’05 Performance Results](https://reader035.fdocuments.net/reader035/viewer/2022062718/56812b3a550346895d8f4ac5/html5/thumbnails/27.jpg)
-12
-10
-8
-6
-4
-2
0
2
4
6
12 13 14 15 16 17 18 19 20 21 22 23 0 1 2 3 4 5 6 7 8 9 10 11
Time (UTC)
Air
T (
C)
OB
RWFS
Eta
GFS
MM5_2
MM5_3
MM5_4
RUC
WRF_2
WRF_3
WRF_4
SN
LOCAL TIME (19 = noon, 06 = midnight)
28 Nov 2005
8C/14F diff2C/4F diff
Air Temperature
Snow
![Page 28: Evaluation of Selected Winter ’04/’05 Performance Results](https://reader035.fdocuments.net/reader035/viewer/2022062718/56812b3a550346895d8f4ac5/html5/thumbnails/28.jpg)
-12
-10
-8
-6
-4
-2
0
12 13 14 15 16 17 18 19 20 21 22 23 0 1 2 3 4 5 6 7 8 9 10 11
Time (UTC)
Td
(C
)
OB
RWFS
Eta
GFS
MM5_2
MM5_3
MM5_4
RUC
WRF_2
WRF_3
WRF_4
SN
LOCAL TIME (19 = noon, 06 = midnight)
28 Nov 2005
6C/11F diff
Dewpoint Temperature
Snow
![Page 29: Evaluation of Selected Winter ’04/’05 Performance Results](https://reader035.fdocuments.net/reader035/viewer/2022062718/56812b3a550346895d8f4ac5/html5/thumbnails/29.jpg)
0
1
2
3
4
5
6
7
8
9
10
12 13 14 15 16 17 18 19 20 21 22 23 0 1 2 3 4 5 6 7 8 9 10 11
Time (UTC)
WS
d (
m/s
)
OB
RWFS
Eta
GFS
MM5_2
MM5_3
MM5_4
RUC
WRF_2
WRF_3
WRF_4
SN
LOCAL TIME (19 = noon, 06 = midnight)
28 Nov 2005
Snow
Wind Speed
![Page 30: Evaluation of Selected Winter ’04/’05 Performance Results](https://reader035.fdocuments.net/reader035/viewer/2022062718/56812b3a550346895d8f4ac5/html5/thumbnails/30.jpg)
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
12 13 14 15 16 17 18 19 20 21 22 23 0 1 2 3 4 5 6 7 8 9 10 11
Time (UTC)
CC
OB
RWFS
Eta
GFS
MM5_2
MM5_3
MM5_4
RUC
WRF_2
WRF_3
WRF_4
OVC
BKN
SCT
FEW
CLR
SN
FEC
LOCAL TIME (19 = noon, 06 = midnight)
28 Nov 2005
Cloud Cover
Snow
![Page 31: Evaluation of Selected Winter ’04/’05 Performance Results](https://reader035.fdocuments.net/reader035/viewer/2022062718/56812b3a550346895d8f4ac5/html5/thumbnails/31.jpg)
0
0.05
0.1
0.15
0.2
0.25
0.3
12 13 14 15 16 17 18 19 20 21 22 23 0 1 2 3 4 5 6 7 8 9 10 11
Time (UTC)
QP
E (
in)
OB
RWFS
Eta
GFS
MM5_2
MM5_3
MM5_4
RUC
WRF_2
WRF_3
WRF_4
SN
LOCAL TIME (19 = noon, 06 = midnight)
28 Nov 2005
Quantitative Precipitation Forecast
Snow
![Page 32: Evaluation of Selected Winter ’04/’05 Performance Results](https://reader035.fdocuments.net/reader035/viewer/2022062718/56812b3a550346895d8f4ac5/html5/thumbnails/32.jpg)
March 13, 2005
• Moderate Snow Event – 4-6” along the E-470 corridor
• Warm air temps before start of snow– Dropped from 11C (52F) to -2C (29F) in 5 hours
• Large variations in parameter predictions– Forecast vs. observations
• Denver International Airport• Ta, Wspd, Cloud Cover and Precipitation
• 00 UTC 13 March 2005 run examined– Captured both start and end times
![Page 33: Evaluation of Selected Winter ’04/’05 Performance Results](https://reader035.fdocuments.net/reader035/viewer/2022062718/56812b3a550346895d8f4ac5/html5/thumbnails/33.jpg)
-8
-6
-4
-2
0
2
4
6
8
10
12
14
16
18
20
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23
Time (UTC)
Air
T (
C)
OB
RWFS
Eta
GFS
MM5_2
MM5_3
MM5_4
RUC
WRF_2
WRF_3
WRF_4
SN
LOCAL TIME (18 = noon, 07 = midnight)
13 March 2005
Air Temperature
Snow
![Page 34: Evaluation of Selected Winter ’04/’05 Performance Results](https://reader035.fdocuments.net/reader035/viewer/2022062718/56812b3a550346895d8f4ac5/html5/thumbnails/34.jpg)
0
1
2
3
4
5
6
7
8
9
10
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23
Time (UTC)
WS
d (
m/s
)
OB
RWFS
Eta
GFS
MM5_2
MM5_3
MM5_4
RUC
WRF_2
WRF_3
WRF_4
SN
LOCAL TIME (18 = noon, 07 = midnight)
13 March 2005
Wind Speed
Snow
![Page 35: Evaluation of Selected Winter ’04/’05 Performance Results](https://reader035.fdocuments.net/reader035/viewer/2022062718/56812b3a550346895d8f4ac5/html5/thumbnails/35.jpg)
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23
Time (UTC)
CC
OB
RWFS
Eta
GFS
MM5_2
MM5_3
MM5_4
RUC
WRF_2
WRF_3
WRF_4
OVC
BKN
SCT
FEW
CLR
SN
LOCAL TIME (18 = noon, 07 = midnight)
13 March 2005
SCT - OVC
Cloud Cover
Snow
![Page 36: Evaluation of Selected Winter ’04/’05 Performance Results](https://reader035.fdocuments.net/reader035/viewer/2022062718/56812b3a550346895d8f4ac5/html5/thumbnails/36.jpg)
actual forecast
Start timeactual forecast
End timeLOCAL TIME (18 = noon, 07 = midnight)
13 March 2005
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23
Time (UTC)
QP
F (
in)
OB
RWFS
Eta
GFS
MM5_2
MM5_3
MM5_4
RUC
WRF_2
WRF_3
WRF_4
Quantitative Precipitation Forecast
![Page 37: Evaluation of Selected Winter ’04/’05 Performance Results](https://reader035.fdocuments.net/reader035/viewer/2022062718/56812b3a550346895d8f4ac5/html5/thumbnails/37.jpg)
Summary/Recommendations
• Large discrepancies between weather models in predicting state weather parameters– All too dry for Td and cloud cover– Low wind speed bias during windy conditions– Overall, no ONE model outperforms => Ensemble
approach key
• Supports probabilistic forecast presentation– Atmosphere is unpredictable– Best approach to present uncertainty to end users?
![Page 38: Evaluation of Selected Winter ’04/’05 Performance Results](https://reader035.fdocuments.net/reader035/viewer/2022062718/56812b3a550346895d8f4ac5/html5/thumbnails/38.jpg)
Thank You!
Questions?