FORECAST ACCURACY VS. FORECAST STABILITYFORECAST STABILITY

Itron Forecasting Brown BagSeptember 18, 2012

PLEASE REMEMBER

» Phones are Muted: In order to help this session run smoothly, your phones are mutedphones are muted.

» Full Screen Mode: To make the presentation portion of the screen larger, press the expand button on the toolbar. g , p pPress it again to return to regular window.

» Feedback: If you need to give feedback to the presenter during the meeting, such as, slow down or need to get the presenters attention for some other reason, use the pull down menu in the seating chart and we will address it right away.

» Questions: If you have questions, please type your question in the Q&A box in the bottom, right corner. We will try to answer as many questions as we canquestions as we can.

2012 BROWN BAG SEMINARS

» Using Conditional Heteroskedastic Variance Models in Load Research Sample Design - March 6, 2012

» Commercial Sales Modeling and Forecasting– June 19, 2012

» Forecast Accuracy vs. Forecast Stability– September 18, 2012p

» Smart Grid Forecasting– December 11, 2012,

All at noon, Pacific TimeAll are recorded and available for review after the session.

WHAT WE DO IN FORECASTINGOperational Forecasting

Financial/BudgetShort-term Forecasting:System OperationsRetail Operations

Energy Trading

Financial/BudgetForecasting

One to three years:Sales forecasts

Capacity PlanningRevenue forecastsVariance analysis Long-term Sales,

Peak and Hourly LoadForecasting

Daily Sales

Hourly Load

DemandResponse Wind

Monthly Revenue

Monthly Sales

PV

Weather

Short Term(e.g., 7 day)

Medium Term(e.g., budget year)

Long Term(e.g., 5‐20 years)( g , y) ( g , g y ) ( g , y )

TRANSMISSION SYSTEM OPERATOR CLIENTS

• Australian Energy Market Operator (AU)• Western Power TSO (AU)• Independent Electricity System Operator (CA)• Independent Electricity System Operator (CA)• Alberta Electricity System Operator (CA)• California ISO (US)• New York ISO (US)• Midwest ISO (US)• PJM (US)( )• ISO New England (US)• ERCOT (US)• BPA (US)• BPA (US)• Southern Company (US)

INDUSTRY LEADING RESEARCH

» PJM – PJM conducted an industry leading study of long-term load forecast drivers This led to a detailed analysisterm load forecast drivers. This led to a detailed analysis of forecast stability. Recommended reading for anyone responsible for developing long-term load forecasts. Go to www.pjm.com and search for:

R i f PJM M d l Ph I L d F t• Review of PJM Models, Phase I: Load Forecast Model Evaluation (Sept, 2010)

• Update on PJM Models Forecast Performance Recommendations (Nov, 2011)

INDUSTRY LEADING RESEARCH » NYISO – the NYISO has advanced the art & practice of

5-minute level load and ramp rate modeling. A summary of their near-term forecast framework can be found atof their near-term forecast framework can be found at www.nyiso.com and search for:

• New Real-time Forecasting Method.ppt (Feb, 2012)

» MIDWEST ISO – the Midwest ISO conducted primary research in the area of Data Filtering and Smoothing to support Near-term Forecasting. The results of thissupport Near term Forecasting. The results of this research is available from ITRON:• A Unified Framework for Near-term and Short-term

S t L d F ti ITRON (M h 2012)System Load Forecasting, ITRON (March, 2012)

INDUSTRY LEADING RESEARCH

» CA ISO – the CAISO is conducting leading edge research on incorporating 5-minute level forecasts of Wind Generation, Solar Plant Generation, and Embedded Solar Generation into their System Operations Forecasting System Look for a presentationOperations Forecasting System. Look for a presentation on these modeling paradigms soon.

SETTING THE STAGE

Slope = 3Slope = 3Slope = 3

We are going to use the HighlySophisticated “Ruler” Model to

G t F t V l iGenerate Forecast Values giventhe most recent loads:

Load(t+h) = Load(t) +hSlope( ) ( ) p

FORECAST VINTAGE #1. GOOD TIMES AHEAD. “AS SOON AS I FILE THIS FORECAST FOR THE RATE CASE I AM GOING TO BUY A NEW CAR”!THE RATE CASE I AM GOING TO BUY A NEW CAR”!

Forecast Vintage #1Forecast Vintage #1Slope = 3

Forecast Vintage #1Forecast Vintage #1

FORECAST VINTAGE #2“HMMM….MUST BE A BLIP IN THE DATA…THE NEW FORDS LOOK COOL”!THE NEW FORDS LOOK COOL”!

Slope = 2.67

Forecast Vintage #1Forecast Vintage #2

FORECAST VINTAGE #3 “YIKES SO MUCH FOR A NEW CAR”!YIKES…SO MUCH FOR A NEW CAR !

Slope = 0.85

Forecast Vintage #1Forecast Vintage #2Forecast Vintage #2Forecast Vintage #3

FORECAST VINTAGE #4“ARE WE AT THE BOTTOM”?ARE WE AT THE BOTTOM ?

Slope = 0.63

Forecast Vintage #1Forecast Vintage #2Forecast Vintage #2Forecast Vintage #3Forecast Vintage #4

FORECAST VINTAGE #5“PHEW!...THINGS ARE GETTING BETTER… BUT I WILL HOLD OFF ON THE NEW CAR”BUT I WILL HOLD OFF ON THE NEW CAR”.

Slope = 1.66

Forecast Vintage #1Forecast Vintage #1Forecast Vintage #2Forecast Vintage #3Forecast Vintage #4Forecast Vintage #5Forecast Vintage #5

“WHAT?!…MY BOSS SAYS THE FORECAST IS TOO HIGH! WHAT DO I DO”?TOO HIGH! WHAT DO I DO ?

XSlope = 1.66 XForecast Vintage #1Forecast Vintage #2Forecast Vintage #2Forecast Vintage #3Forecast Vintage #4Forecast Vintage #5

“BUMMER…THE PUC WANTS TO KNOW WHY THE FORECAST FOR AUG ’09 IS DIFFERENT FROMTHE RATE CASE FILING WHAT DO I TELL THEM”?THE RATE CASE FILING. WHAT DO I TELL THEM”?

Slope = 1.66

Forecast Vintage #1Forecast Vintage #1Forecast Vintage #2Forecast Vintage #3Forecast Vintage #4Forecast Vintage #5Forecast Vintage #5

WHAT DO YOU DO?

» Edit the forecast until my boss is happyC ? ?» Change the model? But how?

» Throw away the last couple of observations» Buy an economic forecast» Buy an economic forecast» Look for a new job

WHAT HAVE WE LEARNED?

» To the audience, forecast instability:• lowers the credibility of the forecast and forecaster• lowers the credibility of the forecast and forecaster• raises the potential for conflict

» To the forecaster, forecast instability leads to questioning of the:• forecast framework, • model specification(s), • forecast drivers, and ,• historical data

FORECAST ACCURACY VS. STABILITY

» Historically, forecast performance metrics were based on forecast accuracyforecast accuracy

» The global financial crisis, along with primary research on very near-term forecasting has raised the possibility y g p ythat blind pursuit of forecast accuracy could lead to forecast instability.I it ibl t h f t d f t» Is it possible to have forecast accuracy and forecast stability during periods of load instability?

FORECAST ACCURACY METRICSIN UNITS OF THE DEPENDENT VARIABLEIN UNITS OF THE DEPENDENT VARIABLE

NORMALIZED MEASURES

NORMALIZED MEASURES

FORECAST ACCURACY IS ABOUT COMPARING FORECASTS TO ACTUALCOMPARING FORECASTS TO ACTUAL.

Forecast Vintage #1Forecast Vintage #2Forecast Vintage #3Forecast Vintage #4Forecast Vintage #4Forecast Vintage #5

FORECAST STABILITY IS ABOUT COMPARING FORECASTS TO FORECASTSFORECASTS TO FORECASTS.

Forecast Vintage #1Forecast Vintage #2Forecast Vintage #3Forecast Vintage #4Forecast Vintage #5Forecast Vintage #5

A MEASURE OF FORECAST STABILITYSMALL VALUES IMPLY STABILITYSMALL VALUES IMPLY STABILITY

MEASURING FORECAST STABILITY

WHAT CREATES FORECAST INSTABILITY?

» Short-term Forecasting( )• Bad Data (e.g. data spikes, missed reads)

• Unexpected regime shifts (e.g. unexpected production outages)production outages)

• Weather forecast uncertainty• Bad model specificationsp

STEPS TO CREATE FORECAST STABILITY

» Short-term Forecasting• Bad Data• Bad Data

- VEE, data smoothing, remove autoregressive terms• Unexpected regime shifts

- Incorporate exogenous drivers, remove autoregressive terms, add autoregressive terms

• Weather forecast uncertainty- Use multiple weather service providers- Use a weighted average weather forecast- Use simple variables: Avg Temp vs Hourly TempUse simple variables: Avg Temp vs Hourly Temp

• Bad model specifications- Build better models

SHORT-TERM: VEE

Data Spike RemovalData Spike Removal

Level Shift Removal

SHORT-TERM: SAVITZKY-GOLAY SMOOTHING

SHORT-TERM: FILTERED & SMOOTHED

SHORT-TERM: MULTIPLE WEATHER SERVICE PROVIDERSWEATHER SERVICE PROVIDERS

SHORT-TERM: AUTOREGRESSIVEOR NOT AUTOREGRESSIVE?OR NOT AUTOREGRESSIVE?

Do you want a load following forecast?orDo you want a forecast that ignores the load drops?

WHAT CREATES FORECAST INSTABILITY?

» Long-term Forecasting( )• Bad Data (e.g. data blips in billing system)

• Unexpected regime shifts (e.g. global financial crisis)• Economic forecast uncertainty• Economic forecast uncertainty• Bad model specifications

STEPS TO CREATE FORECAST STABILITY

» Long-term Forecasting• Bad Data• Bad Data

- Clean the data before model estimation• Unexpected regime shifts

- Remove transient shifts, model permanent shifts• Economic forecast uncertainty

- Use an Index that blends GDP/GMP with Employment/Population

- Use multiple economic forecasts or economic vendors- Use a weighted average economic forecastUse a weighted average economic forecast

• Bad model specifications- Build better models

ALTERNATIVES TO GMP --WEIGHTED ECONOMIC INDEXWEIGHTED ECONOMIC INDEX

GDPGDP

GMP

PInc

Index1de

PopHH

NonMan Emp

ALTERNATIVES TO GMP --SECTOR WEIGHTED ECONOMIC INDEXSECTOR WEIGHTED ECONOMIC INDEX

Industrial

Index2

Res

Com

Sector Weights

CONCLUSIONS

» Forecast Accuracy is important….but now we need to include forecast stability into the discussioninclude forecast stability into the discussion

» For short-term forecasting, the emphasis was on model specification (i.e. neural network versus p (regression)….but now we need to include data filtering and smoothing into our tool setF l t f ti th h i i» For long-term forecasting, the emphasis was on moving away from National level drivers to Regional level drivers (i.e. GDP vs. GMP)….but now we need to think about using Index approaches that combined dollar-based drivers with body count drivers.

BOTTOM LINE DURING PERIODS OF INSTABILITY JUDGMENT IS NEEDEDINSTABILITY JUDGMENT IS NEEDED

?

QUESTIONS? Press *6to ask a question

Remaining 2012 Hands-on Workshops & Meetings

to ask a question

g p g» Fundamentals of Short-Term and Hourly Forecasting

– October 11-12, San Diego» Fundamentals of Sales & Demand Forecasting

– rescheduled to November 7-8, Boston» Forecasting 101

– November 7-9, San Diego» Itron Users' Conference

– October 21-23, San Antonio

For more information and registration: www.itron.com/forecastingworkshops

Contact us at: 1.800.755.9585, 1.858.724.2620 or forecasting@itron comforecasting@itron.com