Theoretical Models and Accuracy As we discussed previously, one of the most useful ways of...
12
Theoretical Models and Accuracy • As we discussed previously, one of the most useful ways of evaluating a theoretical model is to compare the predictions of the model against empirical data • Unfortunately, many industries do not provide you with information on the accuracy of their predictions – how many people improve after taking a drug – how many people actually lose weight by eating Subway sandwiches – how many horoscopes accurately predict the future • As a consequence, it is almost impossible for you, as a consumer, to evaluate the quality of the product
-
date post
21-Dec-2015 -
Category
Documents
-
view
217 -
download
0
Transcript of Theoretical Models and Accuracy As we discussed previously, one of the most useful ways of...
Theoretical Models and Accuracy
• As we discussed previously, one of the most useful ways of evaluating a theoretical model is to compare the predictions of the model against empirical data
• Unfortunately, many industries do not provide you with information on the accuracy of their predictions– how many people improve after taking a drug– how many people actually lose weight by eating Subway
sandwiches– how many horoscopes accurately predict the future
• As a consequence, it is almost impossible for you, as a consumer, to evaluate the quality of the product
• In general, weather forecasters make their predictions based on either mathematical models/simulations of weather systems, past weather conditions, intuitive judgments based on the movement of current weather systems, or “all of the above”
• Unfortunately, weather forecasters do not tell us—the consumers—how accurate their forecasts are. We don’t know what the “track record” is for any weather/news team. (Moreover, they don’t keep records of past forecasts on their websites, thereby preventing consumers from investigating the matter themselves.)
A scientific analysis of the accuracy of weather forecasting
• Students recorded the 5-day forecasts (on Sunday nights) for 3 weeks for the following sources
– WGNTV
– NBC
– CH. 7
– Fox News
– Intellicast
– Chicago Tribune
• Dr Fraley recorded the actual, end-of-the-day weather from an unrelated source: Weather Underground.
• This allows us to compare the predictions made by the different teams with the actual weather.
Average temperature observed, as recorded by Weather Underground, Chicago
Days
Ave
rag
e t
em
pe
ratu
re
2 4 6 8 10 12 14
40
45
50
55
60
65
70
April 5th – April 23rd, 2004
Date Predicted Observed Discrepancy
4/5 46.00 41.00 5.0
4/6 48.50 58.00 -9.5
4/7 43.50 54.00 -10.5
4/8 38.00 50.00 -12.0
4/9 38.00 46.00 -8.0
4/12 35.50 38.00 -2.5
4/13 43.00 40.00 3.0
4/14 47.00 50.00 -3.0
4/15 53.50 62.00 -8.5
4/16 55.00 70.00 -15.0
Average absolute discrepancy: 7.7 degrees~ not too bad, eh? ~
Predictions made by WGN, Channel 9, on Sunday nights
Errors tended to get larger as more days passed between when the prediction was made (Sunday evenings) and the day in question.
Five days later, the average prediction was off by about 10 degrees.
(These numbers are based on WGN TV)
Number of days since prediction
Dis
crep
ancy
be
twee
n pr
edic
ted
and
act
ual t
emp
erat
ure
Predicted Temperature
Act
ua
l Te
mp
era
ture
30 40 50 60 70 80
30
40
50
60
70
80
Predicted TemperatureA
ctu
al T
em
pe
ratu
re
30 40 50 60 70 80
30
40
50
60
70
80
Predicted Temperature
Act
ua
l Te
mp
era
ture
30 40 50 60 70 80
30
40
50
60
70
80
Predicted Temperature
Act
ua
l Te
mp
era
ture
30 40 50 60 70 80
30
40
50
60
70
80
Predicted Temperature
Act
ua
l Te
mp
era
ture
30 40 50 60 70 80
30
40
50
60
70
80
Predicted Temperature
Act
ua
l Te
mp
era
ture
30 40 50 60 70 803
04
05
06
07
08
0
WGNTV | Mean Discrepancy = 6.3
NBC | Mean Discrepancy = 4.2 CH. 7 | Mean Discrepancy = 7
Fox | Mean Discrepancy = 3.9 Intellicast | Mean Discrepancy = 8.9
Chicago Tribune| Mean Discrepancy = 6.2
Date Observed Predicted
4/5 41.00
4/6 58.00 41.00
4/7 54.00 58.00
4/8 50.00 54.00
4/9 46.00 50.00
4/12 38.00 46.00
4/13 40.00 38.00
4/14 50.00 40.00
4/15 62.00 50.00
4/16 70.00 62.00
As an additional comparison, we can generate predictions in another way: based on “day before” temperature
In this case, we’re assuming that the best bet for tomorrow’s temperature is today’s.
Average discrepancy: 7.35
What about broader weather conditions?
• Broader weather conditions– (a) sunny/clear– (b) partly cloudy/mostly cloudy– (c) thunderstorm– (d) rain/showers
• Accuracy in prediction– NBC 73%– Intellicast 80%– Chi. Trib 40%– Day before 71%
Summary
• It appears that the various weather teams do a pretty good job in predicting the temperature– They were only off by approximately 4 – 9 degrees, on average.
– Their ability to make accurate forecasts was lower for extended forecasts.
• In predicting weather conditions (e.g., rain, shine), the various stations were right 40 (Chi. Tribune) – 80% (Intellicast) of the time.
• Nonetheless, you wouldn’t know how good they are without doing the research yourself.
Quiz1. True/False: According to the graphs last week. You can predict a
persons level of Extroversion based on their Astrological Sign.
2. In our in-class exercise on horoscopes, which group felt that their horoscopes were most valid?a) the group with the true horoscopesb) the group with the true and false horoscopes provided fairly
comparable responsesc) the group with the Sun Times horoscopes
3. Why should we care about the pseudosciences in health care?a) People will use them in stead of scientifically supported health
care.b) They take money away from psycologists.c) They might harm people’s souls.
