Sources of Skill and Error in Long Range Columbia River Streamflow Forecasts: A Comparison of the...
1
Sources of Skill and Error in Long Range Columbia River Streamflow Forecasts: A Comparison of the Role of Hydrologic State Variables and Winter Climate Forecasts JP1.2 JISAO/SMA Climate Impacts Group and the Department of Civil and Environmental Engineering, University of Washington Alan F. Hamlet, Andy Wood, Seethu Babu, Dennis P. Lettenmaier Overview In the Columbia River Basin (CRB) in the Pacific Northwest (PNW) skillful winter climate forecasts based on the El Niño Southern Oscillation (ENSO) and Pacific Decadal Oscillation (PDO) are available with lead times of up to about six months (i.e. June 1). Using a hydrologic model and near real time estimates of hydrologic state variables such as soil moisture and snowpack, probabilistic forecasts of streamflow at various basin locations can be produced starting in June (preceding the water year) and proceeding through the snow accumulation season. A useful standard of comparison for these experimental forecasts is a January 1 Ensemble Streamflow Prediction (ESP) forecast, which is comparable in skill and reliability to the statistical streamflow forecasts relating winter snow accumulation to spring and summer streamflow that are used operationally for water management throughout much of the western US. In order to understand the role of the various sources of hydrologic predictability, we generate probabilistic forecasts retrospectively for 50 years using both climate forecasts and estimates of hydrologic state variables on Oct 1, Nov 1, and Dec 1, and compare them, using a quantitative skill metric, to Jan 1 ESP forecasts based solely on hydrologic state variables. Tools and Forecasting Methods The figure below shows a schematic diagram of the forecasting method. Initial snow and soil moisture conditions are assumed to be “perfectly” known at each forecast date. A perfect categorical (warm, neutral, cool) ENSO forecast is also assumed. The VIC hydrologic model (version 4.0.3) was implemented over the Columbia River basin at ¼ degree resolution for these experiments. A schematic of the hydrologic model is shown above. Simulated and observed streamflow at The Dalles, OR (an important checkpoint for water resources management) were used in assessing the skill of the forecasts. Skill Metric Skill metrics that evaluate only the central tendency of a forecast ensemble are generally inadequate for our purposes. Instead we use a more sophisticated metric that both rewards accuracy and punishes “spread” in the forecasts. Note that in this case both the forecast and the climatology are treated as an ensembles as opposed to a single deterministic trace. Skill is defined as: ENSO PDO Run Initialized Hydrologic Model Ensemble Streamflow Forecast Select Temperature and Precipitation Data from Historic Record Associated with Forecast Climate Category Climate Forecast Schematic for Forecasting Experiments Using Resampled Observed Data Elevation (m) The Dalles Role of Oct 1 Initial Soil Moisture Streamflow simulations based on a fixed water year (wet, normal, dry) paired with 50 different initial soil conditions have shown that the initial soil state can effect summer streamflow from April-September by as much as +- 8%. This effect has about the same magnitude in normal or dry years, and is somewhat less in wet years. In dry years, the peak streamflow timing in summer is also shown to be significantly affected by the initial soil state in fall. Normal Year Range =16% of ensemble summer mean Dry Year Streamflow peak is earlier when soils are wet, later when soils are dry Oct Nov Dec Jan ESP Oct Nov Dec Jan ESP Oct Nov Dec Jan ESP Results Long-range forecast is more skillful than Jan 1 ESP forecast Long-range forecast is comparable to Jan 1 ESP forecast Long-range forecast is inferior to Jan 1 ESP forecast A few examples from particular water years to illustrate the nature of the experiments and the performance of the skill metric: -1.00 -0.80 -0.60 -0.40 -0.20 0.00 0.20 0.40 0.60 0.80 1.00 1952 1958 1959 1964 1966 1969 1970 1973 1977 1978 1980 1983 1987 1988 1992 1995 1998 SkillScore O C T ENSO NO V ENSO D EC ENSO JAN ESP -2 -1.5 -1 -0.5 0 0.5 1 1.5 1953 1954 1957 1960 1961 1962 1967 1979 1981 1982 1990 1991 1993 1994 1997 SkillScore O C T E N S O NO V ENSO D EC ENSO JAN ESP -1.5 -1 -0.5 0 0.5 1 1.5 1951 1955 1956 1963 1965 1968 1971 1972 1974 1975 1976 1984 1985 1986 1989 1996 SkillScore O C T ENSO NO V ENSO D EC ENSO JAN ESP Warm ENSO Years ENSO Neutral Years Cool ENSO Years EN SO /PD O (E poch) -1 -0.8 -0.6 -0.4 -0.2 0 0.2 0.4 ENSO /PD O (E poch) E NSO TRANS -1 -0.8 -0.6 -0.4 -0.2 0 0.2 0.4 ENS O TRANS ENSO -1 -0.8 -0.6 -0.4 -0.2 0 0.2 0.4 ENSO ENSO /PD O (Annual) -1 -0.8 -0.6 -0.4 -0.2 0 0.2 0.4 ENSO /PD O (Annual) Changes in Jan 1 ESP Forecast Skill Associated with Climate Information (positive means improved skill) 32 of 48 improved or unchanged 26 of 48 improved or unchanged 35 of 48 improved or unchanged 33 of 48 improved or unchanged Conclusions PDO/ENSO forecasts are estimated to be worth more than $100 million per year in the Columbia basin in terms of hydropower revenue alone (value of forecast), but an objective comparison with a Jan 1 ESP forecasts is useful in understanding the skill and error characteristics of the forecasts in the context of water management. Forecast skill metrics that reward accuracy and punish spread are more appropriate than simple metrics based on the central tendency of the forecasts when the forecasts have different variability than the climatology and there are relatively small shifts in the central tendency of the forecast from year to year. Fall and early winter ENSO-based long-lead forecasts for the Columbia basin (based on resampling methods) typically have a lower skill than January 1 ESP forecasts based on persistence of hydrologic state (soil moisture and snow), but frequently have higher skill than climatology. Prior to December 1 the ENSO based forecasts are considerably less robust than the Jan 1 ESP forecasts, and the climate forecasts play a dominant role in the skill of the forecasts. For ENSO neutral years the skill metrics are questionable due to the distribution of flows within the ensemble, but the skill is generally lower than in warm and cool ENSO years Adding ENSO, ENSO transitions, and interannual PDO climate information to January ESP forecasts improves or leaves the forecast skill unchanged about 70% of the time. An interannual formulation of PDO appears to be preferable to an epochal formulation. Wet Year Range =13% of ensemble summer mean. Summary Results
-
Upload
anis-wright -
Category
Documents
-
view
218 -
download
2
Transcript of Sources of Skill and Error in Long Range Columbia River Streamflow Forecasts: A Comparison of the...
Sources of Skill and Error in Long Range Columbia River Streamflow Forecasts: A Comparison of the Role of Hydrologic State Variables and Winter Climate Forecasts
JP1.2JISAO/SMA Climate Impacts Group and the Department of Civil and Environmental Engineering, University of Washington
Alan F. Hamlet, Andy Wood, Seethu Babu, Dennis P. Lettenmaier
OverviewIn the Columbia River Basin (CRB) in the Pacific Northwest (PNW) skillful winter climate forecasts based on the El Niño Southern Oscillation (ENSO) and Pacific Decadal Oscillation (PDO) are available with lead times of up to about six months (i.e. June 1). Using a hydrologic model and near real time estimates of hydrologic state variables such as soil moisture and snowpack, probabilistic forecasts of streamflow at various basin locations can be produced starting in June (preceding the water year) and proceeding through the snow accumulation season. A useful standard of comparison for these experimental forecasts is a January 1 Ensemble Streamflow Prediction (ESP) forecast, which is comparable in skill and reliability to the statistical streamflow forecasts relating winter snow accumulation to spring and summer streamflow that are used operationally for water management throughout much of the western US. In order to understand the role of the various sources of hydrologic predictability, we generate probabilistic forecasts retrospectively for 50 years using both climate forecasts and estimates of hydrologic state variables on Oct 1, Nov 1, and Dec 1, and compare them, using a quantitative skill metric, to Jan 1 ESP forecasts based solely on hydrologic state variables.
Tools and Forecasting MethodsThe figure below shows a schematic diagram of the forecasting method. Initial snow and soil moisture conditions are assumed to be “perfectly” known at each forecast date. A perfect categorical (warm, neutral, cool) ENSO forecast is also assumed.
The VIC hydrologic model (version 4.0.3) was implemented over the Columbia River basin at ¼ degree resolution for these experiments. A schematic of the hydrologic model is shown above. Simulated and observed streamflow at The Dalles, OR (an important checkpoint for water resources management) were used in assessing the skill of the forecasts.
Skill MetricSkill metrics that evaluate only the central tendency of a forecast ensemble are generally inadequate for our purposes. Instead we use a more sophisticated metric that both rewards accuracy and punishes “spread” in the forecasts. Note that in this case both the forecast and the climatology are treated as an ensembles as opposed to a single deterministic trace. Skill is defined as:
Skill = 1 - [ Σ (forecast - obs)2/N / Σ (climatology - obs)2/M ]
where N is the number of forecast ensemble members and M is the number of climatological observations.
A skill of 0, using this metric is equivalent to the climatology, positive values between 0 and 1 are superior to the climatology, and negative values between 0 and -1 are inferior to the climatology.
ENSO
PDO Run Initialized Hydrologic Model
Ensemble StreamflowForecast
Select Temperature and Precipitation Data from Historic Record Associated with
Forecast Climate Category
ClimateForecast
Schematic for Forecasting Experiments Using Resampled Observed Data
Elevation (m)
The Dalles
Role of Oct 1 Initial Soil MoistureStreamflow simulations based on a fixed water year (wet, normal, dry) paired with 50 different initial soil conditions have shown that the initial soil state can effect summer streamflow from April-September by as much as +- 8%. This effect has about the same magnitude in normal or dry years, and is somewhat less in wet years. In dry years, the peak streamflow timing in summer is also shown to be significantly affected by the initial soil state in fall.
Normal Year
Range =16% of ensemble summer mean
Dry Year
Streamflow peak is earlier when soils are wet, later when soils are dry
Oct Nov
Dec Jan ESP
Oct Nov
Dec Jan ESP
Oct Nov
Dec Jan ESP
Results
Long-range forecast is more skillful than Jan 1 ESP forecast
Long-range forecast is comparable to Jan 1 ESP forecast
Long-range forecast is inferior to Jan 1 ESP forecast
A few examples from particular water years to illustrate the nature of the experiments and the performance of the skill metric:
-1.00
-0.80
-0.60
-0.40
-0.20
0.00
0.20
0.40
0.60
0.80
1.00
19
52
19
58
19
59
19
64
19
66
19
69
19
70
19
73
19
77
19
78
19
80
19
83
19
87
19
88
19
92
19
95
19
98
Ski
ll S
core
OCT ENSO
NOV ENSO
DEC ENSO
JAN ESP
-2
-1.5
-1
-0.5
0
0.5
1
1.5
19
53
19
54
19
57
19
60
19
61
19
62
19
67
19
79
19
81
19
82
19
90
19
91
19
93
19
94
19
97
Ski
ll S
core
OCT ENSO
NOV ENSO
DEC ENSO
JAN ESP
-1.5
-1
-0.5
0
0.5
1
1.5
19
51
19
55
19
56
19
63
19
65
19
68
19
71
19
72
19
74
19
75
19
76
19
84
19
85
19
86
19
89
19
96
Ski
ll S
core
OCT ENSO
NOV ENSO
DEC ENSO
JAN ESP
Warm ENSO Years ENSO Neutral Years Cool ENSO Years
ENSO/PDO (Epoch)
-1
-0.8
-0.6
-0.4
-0.2
0
0.2
0.4
ENSO/PDO (Epoch)
ENSO TRANS
-1
-0.8
-0.6
-0.4
-0.2
0
0.2
0.4
ENSO TRANS
ENSO
-1
-0.8
-0.6
-0.4
-0.2
0
0.2
0.4
ENSO
ENSO/PDO (Annual)
-1
-0.8
-0.6
-0.4
-0.2
0
0.2
0.4
ENSO/PDO (Annual)
Changes in Jan 1 ESP Forecast Skill Associated with Climate Information(positive means improved skill)
32 of 48 improved or unchanged
26 of 48 improved or unchanged
35 of 48 improved or unchanged
33 of 48 improved or unchanged
ConclusionsPDO/ENSO forecasts are estimated to be worth more than $100 million per year in the Columbia basin in terms of hydropower revenue alone (value of forecast), but an objective comparison with a Jan 1 ESP forecasts is useful in understanding the skill and error characteristics of the forecasts in the context of water management.
Forecast skill metrics that reward accuracy and punish spread are more appropriate than simple metrics based on the central tendency of the forecasts when the forecasts have different variability than the climatology and there are relatively small shifts in the central tendency of the forecast from year to year.
Fall and early winter ENSO-based long-lead forecasts for the Columbia basin (based on resampling methods) typically have a lower skill than January 1 ESP forecasts based on persistence of hydrologic state (soil moisture and snow), but frequently have higher skill than climatology. Prior to December 1 the ENSO based forecasts are considerably less robust than the Jan 1 ESP forecasts, and the climate forecasts play a dominant role in the skill of the forecasts. For ENSO neutral years the skill metrics are questionable due to the distribution of flows within the ensemble, but the skill is generally lower than in warm and cool ENSO years
Adding ENSO, ENSO transitions, and interannual PDO climate information to January ESP forecasts improves or leaves the forecast skill unchanged about 70% of the time. An interannual formulation of PDO appears to be preferable to an epochal formulation.
Wet Year
Range =13% of ensemble summer mean.
Summary Results
