Professor: Bob Nau Course content: How to learn from the past

Decision 411: ForecastingDecision 411: Forecasting

Professor: Bob NauProfessor: Bob Nau

Course content:Course content:How to predict the futureHow to predict the futureHow to learn from the pastHow to learn from the past…using data analysis…using data analysis

Who should be interested:Who should be interested:

Anyone on a quantitative career track (financial Anyone on a quantitative career track (financial investments, marketing research, consulting, investments, marketing research, consulting, operations, accounting, econometrics, engineering, operations, accounting, econometrics, engineering, environmental science, policy analysis …)environmental science, policy analysis …)

Anyone who wants more experience in computer Anyone who wants more experience in computer modeling & data analysismodeling & data analysis

Anyone who needs to make decisions based on Anyone who needs to make decisions based on forecasts provided by othersforecasts provided by others

Forecasts are used at every Forecasts are used at every organizational levelorganizational level

Corporate Strategy

FinanceMarketing Accounting

Production, Operations & Supply Chain

Sales

Many numbers…. or one number?

2003 Nobel 2003 Nobel Prize(sPrize(s) in Economics ) in Economics awarded for forecasting methodsawarded for forecasting methods

Robert F. EngleRobert F. Engle“for methods of analyzing economic time series “for methods of analyzing economic time series with timewith time--varying volatility (ARCH)”varying volatility (ARCH)”

Clive W.J. GrangerClive W.J. Granger"for methods of analyzing economic time series "for methods of analyzing economic time series with common trends (with common trends (cointegrationcointegration)”)”

www.nobel.se/economics/laureates/2003/www.nobel.se/economics/laureates/2003/

Recent history (pitfalls of forecasting)Recent history (pitfalls of forecasting)

DJIA to March 2000

1980 1985 1990 1995 2000 20050

1

2

3(X 10000)

Recent historyRecent history

DJIA to March 2000Forecasts (GRW)Lower 95%Upper 95%

1980 1985 1990 1995 2000 20050

1

2

3(X 10000)


DJIA to March 2000Forecasts (GRW)Lower 95%Upper 95%DJIA since March 2000

1980 1985 1990 1995 2000 20050

1

2

3(X 10000)

Today’s agendaToday’s agenda⇒⇒ Course introductionCourse introduction

Forecasting tools & principlesForecasting tools & principles

How to obtain data & move it aroundHow to obtain data & move it around

Statistical graphicsStatistical graphics

Forecasts and confidence intervals: the simplest Forecasts and confidence intervals: the simplest case (mean model)case (mean model)

Betting markets (Betting markets (TradesportsTradesports, etc.), etc.)

Course objectivesCourse objectivesHow to use data to predict the future & aid How to use data to predict the future & aid decisiondecision--makingmaking

Data acquisition and integrationData acquisition and integration

Statistical & graphical data analysisStatistical & graphical data analysis

Regression and other forecasting modelsRegression and other forecasting models

Time series conceptsTime series concepts

Management of forecastingManagement of forecasting

Course mapCourse mapForecasting methods

Statistical Non-statistical

Extrapolative(one variable)

Associative(many variables)

Random walk

Seasonal decomposition

Smoothing

ARIMA

One equation(regression)

Many equations(econometric)

Nonlinear (data mining via neural nets, classification trees, etc.)

Simulation (what-if)

Subjective(expert consensus, field estimates)

We are mainly here

Betting markets

Course outlineCourse outlineWeek 1: Data concepts & simple models: Week 1: Data concepts & simple models:

linear trend & random walk linear trend & random walk

Week 2: Seasonal adjustment & exponential Week 2: Seasonal adjustment & exponential smoothing (smoothing (HW#1 due Tues 9/11HW#1 due Tues 9/11))

Week 3: Regression Week 3: Regression (HW#2 due Tues 9/18(HW#2 due Tues 9/18))

Week 4: More regression (Week 4: More regression (Quiz on Tues 9/25Quiz on Tues 9/25))

Week 5: ARIMA models (Week 5: ARIMA models (HW#3 due Tues 10/2HW#3 due Tues 10/2))

Week 6: Additional topics (automatic, nonlinear…)Week 6: Additional topics (automatic, nonlinear…)

Final project (Final project (due at end of exam week Wed 10/17due at end of exam week Wed 10/17))

ReadingsReadingsMy notes handed out in class & on course web page:My notes handed out in class & on course web page:

faculty.fuqua.duke.edu/~rnau/Decision411CoursePage.htmlfaculty.fuqua.duke.edu/~rnau/Decision411CoursePage.html

PowerpointPowerpoint slides from lecturesslides from lectures

Additional materials on web page, bulletin board, & Additional materials on web page, bulletin board, & CD’sCD’s

Optional stats textbook of your choice Optional stats textbook of your choice

Some forecasting texts are also on reserve in the Some forecasting texts are also on reserve in the Fuqua libraryFuqua library

SoftwareSoftware

StatgraphicsStatgraphics XV (in lab & on your PC)XV (in lab & on your PC)

ExcelExcel

Library databases (Library databases (EconomagicEconomagic, etc.), etc.)

GoogleGoogle

Decision 411 CD’sDecision 411 CD’s

Video files that provide a tour of Video files that provide a tour of StatgraphicsStatgraphics & & EconomagicEconomagic on your own PCon your own PC

View with View with CamtasiaCamtasia Player (included on CD) Player (included on CD)

Hit Hit AltAlt--EnterEnter to toggle the control barto toggle the control bar

Bulletin boardBulletin boardMain course bMain course b--board:board:

mba.fall_1_2007.decision411.forecastingmba.fall_1_2007.decision411.forecasting

Will be used for answers to FAQ’s, additional Will be used for answers to FAQ’s, additional comments on lecture topics, & discussions of statistics comments on lecture topics, & discussions of statistics in the news and in the workplacein the news and in the workplace——check it frequentlycheck it frequently

Feel free to post your own examples of Feel free to post your own examples of good/bad/interesting stats (extra credit for class good/bad/interesting stats (extra credit for class participation!)participation!)

Do Do notnot post any post any assignmentassignment--relatedrelated questions.questions.

EE--mailmail

If you have a question If you have a question for mefor me, send it by , send it by ee--mailmail rather than posting on a brather than posting on a b--board… board…

…but check main b…but check main b--board first to see it has board first to see it has already been asked and answeredalready been asked and answered

Use a descriptive subject line beginning with Use a descriptive subject line beginning with “Forecasting:…”“Forecasting:…”

Grading basisGrading basis

45% homework (3 assignments)45% homework (3 assignments)

15% quiz15% quiz

30% final project30% final project

10% class participation10% class participation

Study group policyStudy group policyWork in teams of 2 (max)Work in teams of 2 (max)

Try to find a partner by FridayTry to find a partner by Friday

OK to team up with someone from other sectionOK to team up with someone from other section

Send me eSend me e--mail if still seeking a partnermail if still seeking a partner

Homework assignmentsHomework assignmentsAssignments will give detailed instructions on Assignments will give detailed instructions on how to obtain the data, import it into how to obtain the data, import it into StatgraphicsStatgraphics, and do much of the analysis., and do much of the analysis.

Your task is to discover the interesting and Your task is to discover the interesting and important patterns and determine a “best” model important patterns and determine a “best” model for purposes of forecasting and/or decision for purposes of forecasting and/or decision making from among a given set of modelsmaking from among a given set of models

Have fun!Have fun!

Homework guidelinesHomework guidelinesAssignments should be submitted in Assignments should be submitted in PowerpointPowerpointform, ideally no more than 15 slides.form, ideally no more than 15 slides.Don’tDon’t worry about fancy fonts, backgrounds, or clip worry about fancy fonts, backgrounds, or clip artart——focus on the important technical issues.focus on the important technical issues.First one or two slides should state your most First one or two slides should state your most important findings, describe your final model, give important findings, describe your final model, give your bottomyour bottom--line forecasts & confidence intervalsline forecasts & confidence intervalsSubsequent slides should document the sequence Subsequent slides should document the sequence of steps by which you reached these conclusions of steps by which you reached these conclusions (data exploration, comparison of models, etc.)(data exploration, comparison of models, etc.)CutCut--andand--paste key reports and graphs from paste key reports and graphs from StatgraphicsStatgraphics and add your specific comments on and add your specific comments on what they tell you about the data.what they tell you about the data.

Final projectFinal projectFinal project may be based on a data set and Final project may be based on a data set and modeling goal of YOUR choicemodeling goal of YOUR choice

Should get started by 5th week of classShould get started by 5th week of class

Alternatively, there will be several “designated Alternatively, there will be several “designated project” options (essentially a fourth homework project” options (essentially a fourth homework assignment)assignment)

Can work in groups of 2 on final project as well Can work in groups of 2 on final project as well as regular homeworkas regular homework

Honor code issuesHonor code issues

You are encouraged to consult your classmates for You are encouraged to consult your classmates for general advice on forecasting concepts and general advice on forecasting concepts and software usesoftware use

Specific details of data analysis assignments should Specific details of data analysis assignments should be discussed only with your studybe discussed only with your study--group partnergroup partner

Don’t post notes on bDon’t post notes on b--board that are at all related to board that are at all related to assignments prior to due datesassignments prior to due dates——send any send any questions to me by equestions to me by e--mail.mail.

Suggestions & examples welcome!Suggestions & examples welcome!

If you are interested in particular forecasting If you are interested in particular forecasting problems or can suggest particular examples problems or can suggest particular examples that might be useful for classroom discussion, that might be useful for classroom discussion, please send me eplease send me e--mailmail (include data if you (include data if you have it)have it)

Exception: no examples from Exception: no examples from gradedgradedassignments in other ongoing courses!assignments in other ongoing courses!

Today’s agendaToday’s agenda

Course introductionCourse introduction

⇒⇒ Forecasting tools & principlesForecasting tools & principles




Betting marketsBetting markets

How can we predict the future?How can we predict the future?Look for Look for statistical patternsstatistical patterns that were stable in the that were stable in the past past and which can be expected to remain stable*and which can be expected to remain stable*

Extrapolate those patterns into the futureExtrapolate those patterns into the future

One important test of assumed patterns is whether One important test of assumed patterns is whether “unexplained” variations (forecast errors) are “unexplained” variations (forecast errors) are independent and identically distributedindependent and identically distributed (“(“i.i.di.i.d.”).”)

* “I have seen the future and it is very much like the present, * “I have seen the future and it is very much like the present, only longer.”only longer.”

−−KehlogKehlog AlbranAlbran, “The Profit”, “The Profit”

Example: Example: i.i.di.i.d. variations around a . variations around a horizontal linehorizontal line

Time Series Plot for X

0 20 40 60 80 100 1200

20

40

60

80

100

X

Time Sequence Plot for XConstant mean = 49.9744

0 20 40 60 80 100 1200

20

40

60

80

100

X

actualforecast95.0% limits

Properties of objects coming off an assembly line might have this pattern.More complex patterns are often reducible to this pattern by suitable transformations.

Example: Example: i.i.di.i.d. variations around a . variations around a trend linetrend line

Time Series Plot for Z

0 20 40 60 80 100 12050

75

100

125

150

175

200

Z

Time Sequence Plot for ZLinear trend = 51.5832 + 0.968143 t

0 20 40 60 80 100 12050

75

100

125

150

175

200

Z


Trends are usually not perfectly linear, and variations around them are usually not i.i.d., but this assumption is often used as a 1st-order approximation for trended data

Example: Example: i.i.di.i.d. . changeschanges in the level of in the level of the series from one period to the nextthe series from one period to the next

Time Series Plot for Y

0 20 40 60 80 100 1200

20

40

60

80

100

Y

Time Sequence Plot for YRandom walk with drift = 0.434184

0 20 40 60 80 100 1200

20

40

60

80

100

Y


This pattern is often seen in financial markets and also in manyphysical processes (e.g. Brownian motion)

Example: stable seasonal patternExample: stable seasonal pattern

RetailxAutoNSA

1992 1994 1996 1998 2000 2002 2004 20061

1.5

2

2.5

3

3.5(X 100000)

RetailxAutoNSAFORECAST

1992 1994 1996 1998 2000 2002 2004 20061

1.5

2

2.5

3

3.5(X 100000)

This pattern is typically seen in retail sales data and in various measures of macroeconomic activity

Example: stable correlations Example: stable correlations among variablesamong variables

age

features

price

sqfeet

tax

Correlations provide a basis for using regression models to predict some variables from others.

TransformationsTransformationsSometimes a stable pattern is not apparent on a Sometimes a stable pattern is not apparent on a graph of the “raw” datagraph of the “raw” data

Transformations of the data (deflation, logging, Transformations of the data (deflation, logging, differencing, seasonal adjustment…) may help differencing, seasonal adjustment…) may help to reveal the underlying patternto reveal the underlying pattern

Ideally the transformed data can be fitted by a Ideally the transformed data can be fitted by a relatively simple modelrelatively simple model

Example: stock pricesExample: stock prices

Pattern: exponential growth curve with 1990’s bubble

LoggedLogged stock pricesstock prices

Natural log transformation linearizes the growth : slope of trend line in logged units is average percentage growth. Dips are more clearly seen to coincide with recessions.

LoggedLogged stock pricesstock prices

Logged indices since 1990

Logged & Logged & differenceddifferenced stock pricesstock prices

Difference of natural log ≈ percent change between periods, which is independently and almost identically distributed.

Variance, i.e., “volatility”, may vary over time.

Time Series Plot for adjusted SP500monthclose

1/80 1/84 1/88 1/92 1/96 1/00 1/04 1/08-0.25

-0.15

-0.05

0.05

0.15

0.25

adju

sted

SP

500m

onth

clos

e

Example: U.S. retail sales (excluding autos)

Pattern: strong nominal growth & seasonal pattern

Deflated and seasonally adjusted sales x-autos

Pattern: real growth accelerated in late ’90’s, flattened after March 2000 peak, dipped in September 2001, ramped up again, but recently…?

What if patterns are not stable?What if patterns are not stable?Trends, seasonality, volatility, etc., may vary in timeTrends, seasonality, volatility, etc., may vary in time

This may limit the amount of past data that should This may limit the amount of past data that should be used for fitting the model (don’t merely use all be used for fitting the model (don’t merely use all data “because it is there”)data “because it is there”)

More sophisticated forecasting models are capable More sophisticated forecasting models are capable of tracking timeof tracking time--varying parametersvarying parameters

Expert opinion can also be used to anticipate Expert opinion can also be used to anticipate changes in patternschanges in patterns

Prices on stock options reveal the “instantaneous Prices on stock options reveal the “instantaneous volatility” of stock prices in the mind of the volatility” of stock prices in the mind of the representative investor.representative investor.

A changing pattern: Housing Starts

Strong seasonal pattern and general upward trend withcyclical variations, but big drop in 2006. When will it rebound?

(A few) Forecasting Principles(A few) Forecasting Principles

Use the most Use the most relevantrelevant & & recentrecent data data

Seek Seek diversediverse & & independentindependent data sourcesdata sources

Let model selection be guided by Let model selection be guided by theorytheory and and domain knowledgedomain knowledge, not just “fit” to past data, not just “fit” to past data

Keep It Keep It SimpleSimple

Test Test the assumptions behind the modelthe assumptions behind the model

ValidateValidate the model on holdthe model on hold--out dataout data

Report Report confidence intervalsconfidence intervals with forecastswith forecasts

The “best” forecasting modelThe “best” forecasting modelIs the one that can be expected to make the Is the one that can be expected to make the SMALLEST ERRORS…SMALLEST ERRORS…

…when predicting the FUTURE* …when predicting the FUTURE*

*not always the same thing as giving the best fit to the past!*not always the same thing as giving the best fit to the past!

Is intuitively reasonableIs intuitively reasonable

Is no more complicated than necessaryIs no more complicated than necessary

Provides insight into trends & causesProvides insight into trends & causes

Can be explained to your boss or clientCan be explained to your boss or client

Forecasting risks (sources of error)Forecasting risks (sources of error)

1.1. IntrinsicIntrinsic risk (random errorrisk (random error——unavoidable, although unavoidable, although may be reduced by more sophisticated modeling)may be reduced by more sophisticated modeling)

2.2. ParameterParameter risk (estimation errorrisk (estimation error——may be reduced may be reduced by collecting more data)by collecting more data)

3.3. ModelModel risk (erroneous assumptionsrisk (erroneous assumptions——often the often the biggest danger!)biggest danger!)

Warning: statistical confidence intervals are based Warning: statistical confidence intervals are based only on estimates of only on estimates of intrinsicintrinsic risk and risk and parameterparameterrisk, not model risk!risk, not model risk!

Intrinsic riskIntrinsic riskEven the best model cannot be expected to make Even the best model cannot be expected to make perfect predictions (“forecasting is hard, especially perfect predictions (“forecasting is hard, especially when it’s about the future…”)when it’s about the future…”)

Intrinsic risk is measured by error statistics such as Intrinsic risk is measured by error statistics such as the the standard error of the model*, mean absolute standard error of the model*, mean absolute error, error, andand mean absolute percentage errormean absolute percentage error

Intrinsic risk can be reduced, in principle, by finding Intrinsic risk can be reduced, in principle, by finding a model that “explains more of the variance” by a model that “explains more of the variance” by making more accurate assumptions and by using making more accurate assumptions and by using more or better “predictor” variablesmore or better “predictor” variables

*Standard error of the model is the error standard deviation adjusted for # coefficients estimated

Parameter riskParameter riskEven if you have the “correct” forecasting model, its Even if you have the “correct” forecasting model, its parameters will not be exactly knownparameters will not be exactly known——they must they must be estimated from available databe estimated from available data

Parameter risk is measured by Parameter risk is measured by standard errors and standard errors and tt--statisticsstatistics of model coefficientsof model coefficients

Parameter risk can be reduced, in principle, by Parameter risk can be reduced, in principle, by using more past data to estimate the modelusing more past data to estimate the model

The “blur of history” problem: older data may be The “blur of history” problem: older data may be “stale” and not reflect current conditions“stale” and not reflect current conditions

Parameter risk is usually a smaller component of Parameter risk is usually a smaller component of forecast error than intrinsic risk or model riskforecast error than intrinsic risk or model risk

Model riskModel riskThis is often the most serious riskThis is often the most serious risk——and its effects and its effects are not taken into account in the calculation of are not taken into account in the calculation of confidence intervalsconfidence intervalsModel risk can be reduced by following good Model risk can be reduced by following good forecasting principles:forecasting principles:Exploratory data analysis to make sure important Exploratory data analysis to make sure important patterns or related variables are not overlookedpatterns or related variables are not overlookedStatistical tests of key assumptionsStatistical tests of key assumptionsOutOut--ofof--sample validation of statistical model sample validation of statistical model (“hold(“hold--out data”, “out data”, “backtestingbacktesting”)”)Use of domain knowledge and expert judgment to Use of domain knowledge and expert judgment to guide model selection and provide “reality checks”guide model selection and provide “reality checks”




⇒⇒ How to obtain data & move it aroundHow to obtain data & move it around




Where to get dataWhere to get data

Internet sources (Internet sources (EconomagicEconomagic, library , library databases, government agencies…)databases, government agencies…)

Your corporate databaseYour corporate database

Trade associations & journalsTrade associations & journals

Econometric consulting firmsEconometric consulting firms

Designed experiments and surveysDesigned experiments and surveys

How to move data aroundHow to move data aroundMost computer programs use their own Most computer programs use their own idiosyncratic “binary” file formats for storing data idiosyncratic “binary” file formats for storing data (word processors, spreadsheets, stat programs, (word processors, spreadsheets, stat programs, database programs…)database programs…)

All programs must also read and write All programs must also read and write text filestext files in in order to communicate with order to communicate with peoplepeople

Hence, different programs can always exchange Hence, different programs can always exchange data data with each otherwith each other in the form of text filesin the form of text files

1 1 charactercharacter of text data = 1 of text data = 1 bytebyte of storageof storage

Text filesText filesMay be either “fixed format” or “delimited”May be either “fixed format” or “delimited”

In a fixed format file, data fields are delineated by In a fixed format file, data fields are delineated by character position within a linecharacter position within a line

xxxxxxxxxx xxxxxxxxxx xxxxxxxxxx xxxxxxxxxx

In a delimited file, data fields are separated by In a delimited file, data fields are separated by delimiting characters (commas, tabs, spaces)delimiting characters (commas, tabs, spaces)

xxxxxxxxxx, , xxxxxxxx, , xxxxxxxxxx, , xxxxxxxxxx, ,

StatgraphicsStatgraphics & Excel can easily read tab& Excel can easily read tab-- or commaor comma--delimited files as well as XLS filesdelimited files as well as XLS files

From From EconomagicEconomagic to to StatgraphicsStatgraphics**Save several series to personal workspaceSave several series to personal workspace

Create Excel file or CSV (commaCreate Excel file or CSV (comma--separatedseparated--value) filevalue) file

Open the file in Excel & clean it up (delete extraneous Open the file in Excel & clean it up (delete extraneous rows, add more descriptive column headings as rows, add more descriptive column headings as variable names)variable names)

Save the cleanedSave the cleaned--up file under a up file under a new namenew name, , CLOSE CLOSE ITIT, and open it in , and open it in StatgraphicsStatgraphics

* See video for details* See video for details

Today’s agendaToday’s agendaCourse introductionCourse introduction



⇒⇒ Statistical graphicsStatistical graphics




Wizards & integrated plotting procedures make Wizards & integrated plotting procedures make charting easycharting easy

Complex patterns in data can be uncovered and Complex patterns in data can be uncovered and communicated by following principles of good communicated by following principles of good graphic designgraphic design

Charts can also be boring, confusing, or Charts can also be boring, confusing, or deceptive if produced thoughtlesslydeceptive if produced thoughtlessly

Tufte’sTufte’s graphical principles*graphical principles*Above all else, Above all else, show the datashow the data

Avoid “Avoid “chartjunkchartjunk”: dark grid lines, false perspective, ”: dark grid lines, false perspective, unintentional optical art, selfunintentional optical art, self--promoting graphicspromoting graphics

Maximize the ratio of data ink to nonMaximize the ratio of data ink to non--data inkdata ink

Mobilize every graphical element, perhaps several Mobilize every graphical element, perhaps several times over, to show the data (e.g., data values times over, to show the data (e.g., data values printed on a bar chart)printed on a bar chart)* * The Visual Display of Quantitative InformationThe Visual Display of Quantitative Information by E. by E. TufteTufte

Charts vs. tablesCharts vs. tablesChartsCharts are most effective when data are numerous are most effective when data are numerous and/or multiand/or multi--dimensionaldimensional

If the data are oneIf the data are one--dimensional and not too dimensional and not too numerous, or if numerical details are important, a numerous, or if numerical details are important, a table table may be better than a chartmay be better than a chart

“A table is nearly always better than a dumb pie “A table is nearly always better than a dumb pie chart; the only worse design than a pie chart is chart; the only worse design than a pie chart is several of them”several of them”

Focus attentionFocus attentionDon’t embed important numbers in sentences of Don’t embed important numbers in sentences of texttext——set them apart in a table or chart.set them apart in a table or chart.

Treat tables & charts as “paragraphs”, and include Treat tables & charts as “paragraphs”, and include them in the narrative at the appropriate pointsthem in the narrative at the appropriate points

Annotate charts with appropriate commentsAnnotate charts with appropriate comments

Maximize data density: “graphs can be shrunk way Maximize data density: “graphs can be shrunk way down” so that more than one will fit on a page or down” so that more than one will fit on a page or slideslide

Excel & Excel & StatgraphicsStatgraphics tipstipsEmbed small, wellEmbed small, well--labeled, welllabeled, well--chosen charts & chosen charts & tables in your reports tables in your reports

Make points and lines thick enough to “show the Make points and lines thick enough to “show the data”data”

Suppress gridlines where not neededSuppress gridlines where not needed

Use an appropriate chart type (e.g., line plots for Use an appropriate chart type (e.g., line plots for time series, time series, scatterplotsscatterplots for crossfor cross--sectional data, sectional data, bar charts or tables rather than pie charts)bar charts or tables rather than pie charts)

Names and units of variables should be clearly Names and units of variables should be clearly shown in titles, legends, and/or axis labelsshown in titles, legends, and/or axis labels

Today’s agendaToday’s agendaCourse introductionCourse introduction




⇒⇒ Forecasts and confidence intervals: the Forecasts and confidence intervals: the simplest case (mean model)simplest case (mean model)


Consider the following time series:Consider the following time series:

Time Series Plot for X

X

0 4 8 12 16 2050

70

90

110

130

150

170

How to forecast?How to forecast?If you have reason to believe the observations are If you have reason to believe the observations are statistically independentstatistically independent and and identically distributedidentically distributed, , with with no trend*no trend*, the appropriate forecasting model is , the appropriate forecasting model is the MEAN modelthe MEAN model

Just predict that future observations will equal the Just predict that future observations will equal the meanmean of the past valuesof the past values

*These assumptions might be based on domain knowledge, or else they could be tested by comparing alternative models and looking at autocorrelations, etc..

Stats review: “population” statisticsStats review: “population” statisticsXX = random variable = random variable NN = size of entire population (possibly infinite)= size of entire population (possibly infinite)nn = size of a finite sample= size of a finite sample

The population (“true”) mean The population (“true”) mean μμ is the average of the is the average of the all values in the population:all values in the population:

The population variance The population variance σσ22 is the average squared is the average squared deviation from the true mean:deviation from the true mean:

The population standard deviation The population standard deviation σσ is the square root is the square root of the population variance, i.e., the “root mean of the population variance, i.e., the “root mean squared” deviation from the true mean.squared” deviation from the true mean.

1Ni ix Nμ ==∑

22 1( )N

ii xN

μσ = −

=∑

Stats review: “sample” statisticsStats review: “sample” statisticsIn forecasting applications, In forecasting applications, we never observe the we never observe the whole population.whole population. The problem is to forecast from a The problem is to forecast from a small sample. Hence statistics such as means and small sample. Hence statistics such as means and standard deviations must be standard deviations must be estimatedestimated with with errorerror..

The The sample meansample mean is the average of the all values in is the average of the all values in the sample:the sample:

The The sample variancesample variance ss22 is the average squared is the average squared deviation from the deviation from the samplesample mean, except with a factor mean, except with a factor of of nn−−11 rather than rather than nn in the denominator:in the denominator:

…and the …and the sample standard deviationsample standard deviation is its square root, is its square root, ss

1ni iX x n==∑

22 1( )

1

nii x X

sn

= −=

−∑

Sample statistics, continuedSample statistics, continuedWhy the factor of Why the factor of nn−−11? This corrects for ? This corrects for the fact that fact that

the mean has been estimated from the same sample, the mean has been estimated from the same sample, which “fudges” it in a direction that makes the mean which “fudges” it in a direction that makes the mean squared deviation around it less than it ought to be.squared deviation around it less than it ought to be.

Technically we say that a “degree of freedom for error” Technically we say that a “degree of freedom for error” has been used up by calculating the sample mean has been used up by calculating the sample mean from the same data.from the same data.

The correct adjustment to get an “unbiased” estimate The correct adjustment to get an “unbiased” estimate of the true variance is to divide by the number of of the true variance is to divide by the number of degrees of freedom, not the number of data pointsdegrees of freedom, not the number of data points

1ni iX x n==∑

In ExcelIn Excel

Population mean = AVERAGE (Population mean = AVERAGE (xx11, …, …xxNN))Population variance = VARP (Population variance = VARP (xx11, …, …xxNN))Population std. dev. = STDEVP(Population std. dev. = STDEVP(xx11, …, …xxNN))

Sample mean = AVERAGE (Sample mean = AVERAGE (xx11, …x, …xnn))Sample variance = VAR (Sample variance = VAR (xx11, …x, …xnn))Sample std. dev. = STDEV (Sample std. dev. = STDEV (xx11, …x, …xnn))

Why Why squared squared error?error?Why should we measure variability in terms of Why should we measure variability in terms of average average squaredsquared deviations instead of average deviations instead of average absoluteabsolute deviations around a central value?deviations around a central value?Squared error has a lot of nice properties:Squared error has a lot of nice properties:

The central value around which average The central value around which average squared squared deviations are minimized is the deviations are minimized is the meanmean, so by attempting , so by attempting to minimize squared deviations we are implicitly to minimize squared deviations we are implicitly calculating means.calculating means.From a decisionFrom a decision--theoretic viewpoint, large errors theoretic viewpoint, large errors usually have proportionally worse consequences than usually have proportionally worse consequences than small errors, hence squared error is more small errors, hence squared error is more representative of economic consequences of error.representative of economic consequences of error.Variances and Variances and covariancescovariances play a key role in normal play a key role in normal distribution theory & regression analysis.distribution theory & regression analysis.

Standard error of the meanStandard error of the mean

This is the estimated standard deviation of the This is the estimated standard deviation of the error error that we would make in using the sample meanthat we would make in using the sample meanas an estimate of the true mean as an estimate of the true mean μμ , if we repeated this , if we repeated this

exercise with other independent samples of size exercise with other independent samples of size nn. .

It measures the It measures the precisionprecision of our estimate of the of our estimate of the (unknown) true mean from a limited sample of data.(unknown) true mean from a limited sample of data.

As As nn gets larger, gets larger, SESEmeanmean gets smaller and the gets smaller and the distribution of errors becomes normal*distribution of errors becomes normal*

*Central Limit Theorem*Central Limit Theorem

meansSE

n=

X

Standard Standard deviationdeviation or standard or standard errorerror??

The term “standard deviation” (usually) refers to the The term “standard deviation” (usually) refers to the actual actual rootroot--meanmean--squared deviation of a given squared deviation of a given population or sample around its meanpopulation or sample around its mean

The term “standard error” refers to the The term “standard error” refers to the expectedexpectedrootroot--meanmean--squared deviation of an estimate or squared deviation of an estimate or forecast around the true value under repeated forecast around the true value under repeated samplingsamplingThus, a standard error is the “standard deviation of Thus, a standard error is the “standard deviation of the error” in estimating or forecasting something the error” in estimating or forecasting something

Forecasting with the mean modelForecasting with the mean modelLet denote a Let denote a forecastforecast of of xxn+n+11 based on data based on data observed up to period observed up to period nnIf If xxn+n+11 is assumed to be independently drawn is assumed to be independently drawn from the same population as the sample from the same population as the sample xx11, …, , …, xxnn, , then the forecast that minimizes mean squared then the forecast that minimizes mean squared error is simply the sample mean:error is simply the sample mean:

Now, what is the standard deviation of the Now, what is the standard deviation of the error error we can expect to make in using as awe can expect to make in using as a

1+nx̂

Xxn =+1ˆ

X

Standard error of the forecastStandard error of the forecastThe The standard error of the forecaststandard error of the forecast has two has two components:components:

2 2 11fcst meanSE s SE s n= + = +

This term measures the intrinsic risk

(“noise” in the data)This term measures the parameter risk

(error in estimating the “signal” in the data)

Note that variances, rather than standard deviations, are additive

For the mean model, the result is that the forecast standard error is slightly larger than the sample

standard deviation

Confidence intervals for forecastsConfidence intervals for forecastsA point forecast should always be accompanied by a A point forecast should always be accompanied by a confidence intervalconfidence interval to indicate its accuracy… but to indicate its accuracy… but what what isis a confidence interval??a confidence interval??

An x% confidence interval is an interval calculated An x% confidence interval is an interval calculated by a by a rulerule which has the property that the interval will which has the property that the interval will cover the true value x% of the time under cover the true value x% of the time under simulatedsimulatedconditions, conditions, assuming the model is correctassuming the model is correct..

Loosely speakingLoosely speaking, there is an x% chance that , there is an x% chance that your your data will fall in data will fall in youryour x% confidence intervalx% confidence interval——but but only if your model and its underlying assumptions only if your model and its underlying assumptions are correct! (This is why we test assumptions.)are correct! (This is why we test assumptions.)

Confidence interval = Confidence interval = point forecast point forecast ±± tt standard errorsstandard errors

If the distribution of forecast errors is assumed to be If the distribution of forecast errors is assumed to be normalnormal, a , a 95% confidence interval95% confidence interval for the forecast isfor the forecast is

…where is the critical value of the …where is the critical value of the “Student’s “Student’s tt” distribution” distribution** with a tail probability of .05 with a tail probability of .05 and and nn−−11 “degrees of freedom” “degrees of freedom”

In Excel, = TINV(.05, In Excel, = TINV(.05, nn−−1)1)

fcstnn SEtx 1051 −+ ± ,.ˆ

105 −nt ,.

105 −nt ,.

*discovered by W.S. Gossett of Guinness Brewery

en.wikipedia.org/wiki/William_Sealey_Gosset

105 −nt ,.

tt vs. normal distributionvs. normal distribution

The The tt distribution is the distribution ofdistribution is the distribution of

which is the which is the number of standard errors by which number of standard errors by which the sample mean deviates from the true meanthe sample mean deviates from the true meanwhen the standard deviation of the population is when the standard deviation of the population is unknown.unknown.

meanSEX )( μ−

The The tt distribution resembles a standard normal (distribution resembles a standard normal (zz) ) distribution but with “fatter tails” for small distribution but with “fatter tails” for small nn

Normal vs. t: much difference?

-4 -3 -2 -1 0 1 2 3 4

Normal t with 20 df t with 10 df t with 5 df

# standard errors # standard errors to use for calculating to use for calculating confidence intervals is confidence intervals is very similar very similar forfor normal normal

and and tt distributions except for very low distributions except for very low d.fd.f. or very . or very high confidencehigh confidence

4.5874.5873.5813.5813.1693.1692.2282.2281.8121.81210103.8503.8503.1533.1532.8452.8452.0862.0861.7251.72520203.4963.4962.9372.9372.6782.6782.0092.0091.6761.67650503.3903.3902.8712.8712.6262.6261.9841.9841.6601.6601001003.3403.3402.8382.8382.6012.6011.9721.9721.6531.6532002003.2913.2912.8072.8072.5762.5761.9601.9601.6451.645Normal Normal

99.9%99.9%99.5%99.5%99.0%99.0%95.0%95.0%90.0%90.0%d.fd.f..Confidence level (2Confidence level (2--sided)sided)

Empirical rules of thumbEmpirical rules of thumbFor For n n ≈≈ 20 or more, the critical 20 or more, the critical tt value is value is approximately 2, so the approximately 2, so the ““empiricalempirical”” 95% CI is 95% CI is roughly the point forecast roughly the point forecast plus or minus two plus or minus two standard errors, standard errors, howeverhowever……

A prediction interval that covers 95% of the data is A prediction interval that covers 95% of the data is often often too widetoo wide to be managerially usefulto be managerially useful——50% (a 50% (a ““coin flipcoin flip””) or 80% might be easier for a manager to ) or 80% might be easier for a manager to understandunderstand

A 50% confidence interval is roughly A 50% confidence interval is roughly plus or minus plus or minus twotwo--thirds of a standard errorthirds of a standard error

Example, continuedExample, continued

Time series X (Time series X (nn=20*, =20*, d.f.d.f. =19**): =19**):

114, 126, 123, 112, 68, 116, 50, 108, 163, 79114, 126, 123, 112, 68, 116, 50, 108, 163, 7967, 98, 131, 83, 56, 109, 81, 61, 90, 9267, 98, 131, 83, 56, 109, 81, 61, 90, 92

Statistics: 96.35, 28.96X s= =

48.620/96.28 ==meanSE

30,100:parametersTrue* == σμ

68.2948.696.28 22 =+=fcstSE

Confidence intervals for predictionsConfidence intervals for predictions

05 19 2 093. ,*t .=

Exact 95% CI* = 96.35 Exact 95% CI* = 96.35 ±± 2.093 2.093 ×× 29.6829.68= [34.2, 158.5]= [34.2, 158.5]

Exact 50% CI** = 96.35 Exact 50% CI** = 96.35 ±± 0.688 0.688 ×× 29.6829.68= = [77.8, 114.9][77.8, 114.9]

5 19 0 688. ,**t .=

StatgraphicsStatgraphics output: mean modeloutput: mean modelTime Sequence Plot for X

Constant mean = 96.35

X


0 10 20 30 4025

50

75

100

125

150

175

StatgraphicsStatgraphics output: mean modeloutput: mean modelTime Sequence Plot for X

Constant mean = 96.35

X


0 10 20 30 4025

50

75

100

125

150

175

A 50% confidence interval is 1/3 the width of a 95% A 50% confidence interval is 1/3 the width of a 95% confidence interval.confidence interval.

What if there’s really a trend?What if there’s really a trend?Time Sequence Plot for XLinear trend = 114.611 + -1.7391 t

X


0 10 20 30 4025

50

75

100

125

150

175

That’s a different modeling assumption, and it leads to That’s a different modeling assumption, and it leads to very different forecasts and confidence intervals.very different forecasts and confidence intervals.

Actually, t=1.61 for slope coefficient, so this model would be rejected at .05 level of significance.

Yes, it’s simple, but...Yes, it’s simple, but...The mean model is the foundation for more The mean model is the foundation for more sophisticated models we will encounter later sophisticated models we will encounter later (random walk, regression, ARIMA)(random walk, regression, ARIMA)

It has the same generic features:It has the same generic features:→→ A A coefficientcoefficient to be estimatedto be estimated→→ A A standard errorstandard error for the coefficientfor the coefficient

that reflects parameter riskthat reflects parameter risk→→ A A forecast standard errorforecast standard error that that

reflects intrinsic risk & parameter riskreflects intrinsic risk & parameter risk→→ …and model risk too!…and model risk too!







⇒⇒ Betting marketsBetting markets

MarketMarket--based forecastingbased forecastingBetting markets are often an efficient way to Betting markets are often an efficient way to aggregate diverse opinions (and to share risks… or aggregate diverse opinions (and to share risks… or have fun)have fun)

Probabilistic forecasts derived from contract prices Probabilistic forecasts derived from contract prices are often wellare often well--calibratedcalibrated

Caveats: markets don’t Caveats: markets don’t alwaysalways workwork——may exhibit may exhibit “herding” or distortions when bettors lack “herding” or distortions when bettors lack independent information or have highly correlated independent information or have highly correlated financial or emotional stakes in eventsfinancial or emotional stakes in events

Some applications are controversial (e.g. “terrorism Some applications are controversial (e.g. “terrorism futures”)futures”)

Forecasting U.S. Open men’s tennis Forecasting U.S. Open men’s tennis champion via a betting marketchampion via a betting market

‘Outright winner” price quotes on Tradesports.com at 5:00pm EDT, August 30: Roger Federer’s estimated probability of winning is around 65%

Forecast for women’s championForecast for women’s champion

Maria Sharapova’s probability of winning was estimated to be 20%

Forecasting the baseball world Forecasting the baseball world series championseries champion

Boston Red Sox are at around 20%

Recap of today’s topicsRecap of today’s topics





•• Forecasts and confidence intervals: the simplest Forecasts and confidence intervals: the simplest case (mean model)case (mean model)

•• Betting marketsBetting markets

Professor: Bob Nau Course content: How to learn from the past

Documents

Transcript of Professor: Bob Nau Course content: How to learn from the past