Most popular “traditional statistics”: ETS, Bias Problem: what does the ETS tell us ?
description
Transcript of Most popular “traditional statistics”: ETS, Bias Problem: what does the ETS tell us ?
![Page 1: Most popular “traditional statistics”: ETS, Bias Problem: what does the ETS tell us ?](https://reader036.fdocuments.net/reader036/viewer/2022062409/56814af2550346895db802bd/html5/thumbnails/1.jpg)
Bias Adjusted Precipitation Scores
Fedor Mesinger
NOAA/Environmental Modeling Centerand
Earth System Science Interdisciplinary Center(ESSIC), Univ. Maryland, College Park, MD
VX-Intercompare MeetingBoulder, 20 February 2007
![Page 2: Most popular “traditional statistics”: ETS, Bias Problem: what does the ETS tell us ?](https://reader036.fdocuments.net/reader036/viewer/2022062409/56814af2550346895db802bd/html5/thumbnails/2.jpg)
Most popular “traditional statistics”:
ETS, Bias
Problem: what does the ETS tell us ?
![Page 3: Most popular “traditional statistics”: ETS, Bias Problem: what does the ETS tell us ?](https://reader036.fdocuments.net/reader036/viewer/2022062409/56814af2550346895db802bd/html5/thumbnails/3.jpg)
“The higher the value, the better the model skill is for the particular threshold”
(a recent MWR paper)
![Page 4: Most popular “traditional statistics”: ETS, Bias Problem: what does the ETS tell us ?](https://reader036.fdocuments.net/reader036/viewer/2022062409/56814af2550346895db802bd/html5/thumbnails/4.jpg)
Example:Three models, ETS, Bias, 12 months, “Western Nest”
Is the green model loosing to red because of a bias penalty?
![Page 5: Most popular “traditional statistics”: ETS, Bias Problem: what does the ETS tell us ?](https://reader036.fdocuments.net/reader036/viewer/2022062409/56814af2550346895db802bd/html5/thumbnails/5.jpg)
What can one do ?
![Page 6: Most popular “traditional statistics”: ETS, Bias Problem: what does the ETS tell us ?](https://reader036.fdocuments.net/reader036/viewer/2022062409/56814af2550346895db802bd/html5/thumbnails/6.jpg)
BIAS NORMALIZED PRECIPITATION SCORES
Fedor Mesinger1 and Keith Brill2
1NCEP/EMC and UCAR, Camp Springs, MD2NCEP/HPC, Camp Springs, MD
J12.617th Prob. Stat. Atmos. Sci.; 20th WAF/16th NWP (Seattle AMS, Jan. ‘04)
![Page 7: Most popular “traditional statistics”: ETS, Bias Problem: what does the ETS tell us ?](https://reader036.fdocuments.net/reader036/viewer/2022062409/56814af2550346895db802bd/html5/thumbnails/7.jpg)
Two methods of the adjustment for bias(“Normalized” not the best idea)
1. dHdF method: Assume incremental change in hits per incremental change in bias is proportional to the “unhit” area, O-
H
Objective: obtain ETS adjusted to unit bias, to show the model’s accuracy in placing
precipitation(The idea of the adjustment to unit bias to arrive at placement accuracy:
Shuman 1980, NOAA/NWS Office Note)
2. Odds Ratio method: different objective
![Page 8: Most popular “traditional statistics”: ETS, Bias Problem: what does the ETS tell us ?](https://reader036.fdocuments.net/reader036/viewer/2022062409/56814af2550346895db802bd/html5/thumbnails/8.jpg)
O
H
a
b
c
d
F
Forecast, Hits, and Observed (F, H, O) area, or number of model grid boxes:
![Page 9: Most popular “traditional statistics”: ETS, Bias Problem: what does the ETS tell us ?](https://reader036.fdocuments.net/reader036/viewer/2022062409/56814af2550346895db802bd/html5/thumbnails/9.jpg)
dHdF method, assumption:
can be solved;
a function H (F) obtained that satisfies the three
requirements:€
dHdF
= a (O − H ), a = const,
![Page 10: Most popular “traditional statistics”: ETS, Bias Problem: what does the ETS tell us ?](https://reader036.fdocuments.net/reader036/viewer/2022062409/56814af2550346895db802bd/html5/thumbnails/10.jpg)
• Number of hits H -> 0 for F -> 0;
• The function H(F) satisfies the
known value of H for the model’s F,
the pair denoted by Fb, Hb, and,
• H(F) -> O as F increases
![Page 11: Most popular “traditional statistics”: ETS, Bias Problem: what does the ETS tell us ?](https://reader036.fdocuments.net/reader036/viewer/2022062409/56814af2550346895db802bd/html5/thumbnails/11.jpg)
![Page 12: Most popular “traditional statistics”: ETS, Bias Problem: what does the ETS tell us ?](https://reader036.fdocuments.net/reader036/viewer/2022062409/56814af2550346895db802bd/html5/thumbnails/12.jpg)
West
EtaGFS
NMM
Bias adjusted eq. threats
![Page 13: Most popular “traditional statistics”: ETS, Bias Problem: what does the ETS tell us ?](https://reader036.fdocuments.net/reader036/viewer/2022062409/56814af2550346895db802bd/html5/thumbnails/13.jpg)
A downside: if Hb is close to Fb, or to O,it can happen that
dH/dF > 1 for F -> 0
Physically unrealistic !
Reasonableness requirement:
€
Hb ≤ O(1−e−Fb /O )
![Page 14: Most popular “traditional statistics”: ETS, Bias Problem: what does the ETS tell us ?](https://reader036.fdocuments.net/reader036/viewer/2022062409/56814af2550346895db802bd/html5/thumbnails/14.jpg)
“dHdM”method:
O
H
a
b
c
d
F
Assume as F is increased by dF, ratio of the infinitesimal increase in H, dH, and that in
false alarms dM=dF-dH, is proportional to the yet unhit area:
![Page 15: Most popular “traditional statistics”: ETS, Bias Problem: what does the ETS tell us ?](https://reader036.fdocuments.net/reader036/viewer/2022062409/56814af2550346895db802bd/html5/thumbnails/15.jpg)
€
dH
dM= b(O − H)
€
b = const
One obtains
( Lambertw, or ProductLog in Mathematica,is the inverse function of
€
z = wew )
€
H(F) = O −1
blambertw bOeb(O−F )
( )
![Page 16: Most popular “traditional statistics”: ETS, Bias Problem: what does the ETS tell us ?](https://reader036.fdocuments.net/reader036/viewer/2022062409/56814af2550346895db802bd/html5/thumbnails/16.jpg)
H (F) now satisfies the additional requirement:
dH/dF never > 1
![Page 17: Most popular “traditional statistics”: ETS, Bias Problem: what does the ETS tell us ?](https://reader036.fdocuments.net/reader036/viewer/2022062409/56814af2550346895db802bd/html5/thumbnails/17.jpg)
20 40 60 80 100 120 140
F
20
40
60
80
100
120H
H(F)
H = O
H = F
Fb , Hb
dHdF method
![Page 18: Most popular “traditional statistics”: ETS, Bias Problem: what does the ETS tell us ?](https://reader036.fdocuments.net/reader036/viewer/2022062409/56814af2550346895db802bd/html5/thumbnails/18.jpg)
H(F)
H = O
H = F
Fb , Hb
20 40 60 80 100 120 140F
20
40
60
80
100
120H
dHdM method
![Page 19: Most popular “traditional statistics”: ETS, Bias Problem: what does the ETS tell us ?](https://reader036.fdocuments.net/reader036/viewer/2022062409/56814af2550346895db802bd/html5/thumbnails/19.jpg)
Results for the two “focus cases”,dHdM method
(Acknowledgements: John Halley Gotway, data; Dušan Jović, code and
plots)
![Page 20: Most popular “traditional statistics”: ETS, Bias Problem: what does the ETS tell us ?](https://reader036.fdocuments.net/reader036/viewer/2022062409/56814af2550346895db802bd/html5/thumbnails/20.jpg)
5/13 Case dHdM
wrf2capswrf4ncarwrf4ncep
![Page 21: Most popular “traditional statistics”: ETS, Bias Problem: what does the ETS tell us ?](https://reader036.fdocuments.net/reader036/viewer/2022062409/56814af2550346895db802bd/html5/thumbnails/21.jpg)
6/01 Case
dHdM
wrf2capswrf4ncarwrf4ncep
![Page 22: Most popular “traditional statistics”: ETS, Bias Problem: what does the ETS tell us ?](https://reader036.fdocuments.net/reader036/viewer/2022062409/56814af2550346895db802bd/html5/thumbnails/22.jpg)
Impact, in relative terms, for the two cases is small, because the biases of the three models
are so similar !
![Page 23: Most popular “traditional statistics”: ETS, Bias Problem: what does the ETS tell us ?](https://reader036.fdocuments.net/reader036/viewer/2022062409/56814af2550346895db802bd/html5/thumbnails/23.jpg)
One more case, for good measure:
![Page 24: Most popular “traditional statistics”: ETS, Bias Problem: what does the ETS tell us ?](https://reader036.fdocuments.net/reader036/viewer/2022062409/56814af2550346895db802bd/html5/thumbnails/24.jpg)
5/25 Case dHdM
wrf2capswrf4ncarwrf4ncep
![Page 25: Most popular “traditional statistics”: ETS, Bias Problem: what does the ETS tell us ?](https://reader036.fdocuments.net/reader036/viewer/2022062409/56814af2550346895db802bd/html5/thumbnails/25.jpg)
Comment:
Scores would have generally been higher had the verification been done on grid squares
greater than ~4 km
This would have amounted to a poor-person’s version of “fuzzy” methods !