Comparison of 1994-1997 CA Evaluation Protocols, CA Framework, IPMVP and CPUC Policy Manual* A preface to group discussion

*In terms of how they define program gross energy and demand EM&V requirements

Mary Sutter & Tim CaulfieldEquipoise Consulting Inc.October 27, 2004

2

Discussion Overview

Purpose of this presentation Overview of some documents used in California

evaluation of energy efficiency programs CPUC Energy Efficiency Policy Manual Version 2 The California Evaluation Framework Protocols and Procedures for the Verification of Costs,

Benefits, and Shareholder Earnings from Demand-Side Management Programs (The “Protocols”)

International Performance Measurement and Verification Protocol - Concepts and Options for Determining Water and Energy Savings (IPMVP)

Comparisons Group Discussion

3

Purpose of this Presentation

CALMAC is the appropriate venue for substantive discussion on evaluation

Energy efficiency program evaluation in California in a state of flux

New evaluation protocols are planned Through review of past / present documents,

determine what are the most crucial questions to discuss at this point in time

Narrow scope for now (Gross energy and demand impacts)

4

Energy Efficiency Policy Manual Purpose: Sets the policy rules

in the development and evaluation of energy efficiency (EE) programs in California.

Broad set of objectives that evaluation must meet.

Specific components indicated with supporting evidence required if not including in evaluation.

5

Policy Manual The objectives below must have strong supporting arguments for omission from evaluation plan

Policies for Programs with Energy Impacts Evaluation must measure the level of energy and peak demand

savings achieved by a program. Measure Cost-effectiveness Up-front market assessments and baseline analysis (especially

for new programs) Ongoing feedback, and corrective and constructive guidance

regarding the implementation of the program Measure indicators of the effectiveness of specific programs,

including testing of the assumptions that underlie the program theory and approach

Assess overall levels of performance and success of programs Inform decisions regarding compensation and final payments Help assess whether there is a continuing need for the program

6

Policy Manual These components are required in the evaluation plan

Baseline Information Baseline data upon which to base energy savings measurement Perform study if none available or prove why cannot do study

Energy Efficiency Measure Information Description of EE measures in program Includes assumptions about important variables and unknowns

M&V Approach Reference appropriate IPMVP option Describe deviation from IPMVP Schedule for acquiring project-specific data

Evaluation Approach Questions to be answered through evaluation Evaluation tasks / activities Describe how evaluation will meet all policy objectives

7

The California Evaluation Framework Purpose: The California

Evaluation Framework (Framework) provides a consistent, systemized, cyclic approach for planning and conducting evaluations of California’s energy efficiency and resource acquisition programs.

The primary purpose of impact evaluation is to obtain the most accurate and unbiased estimate of energy and demand savings due to a program.

Gross savings are calculated from program participants relative to their prior participation usage.

8

Framework – Gross Savings

Billing Analysis path Based on statistical principals Multiple methods / regression models

Engineering Analysis path Based on basic rules of physics Simple engineering models or building energy

simulations M&V incorporated into field data collection (IPMVP)

9

Framework – Gross Savings Issues Billing Analysis

Difficult / impossible for evaluation of 3rd party programs.

Finding a non-participant group not effected by any EE program (market noise)

Engineering Analysis The most uncertain parameter may be the most

expensive to obtain Cannot collect pre-retrofit measurements in many

cases

10

IPMVP

Purpose: The IPMVP provides an overview of current best practice techniques available for verifying results of energy efficiency, water efficiency, and renewable energy projects.

This document can help in the selection of the M&V approach that best matches: i) project costs and savings magnitude ii) technology-specific requirements iii) risk allocation between buyer and

seller (i.e., which party is responsible for installed equipment performance and which party is responsible for achieving long term energy savings).

11

IPMVP M&V – the process of determining savings using one of the four IPMVP options Option A

Engineering calculations with some parameters in algorithms stipulated and other parameters onsite measured

Option B Engineering calculations with all parameters in algorithm onsite

measured Option C

Whole facility analysis with billing or metered data – from simple pre/post comparison to regression analysis on a single building over time

Option D Computer simulation calibrated with hourly or monthly utility

billing data and/or end use metering

12

“Protocols”

Purpose: Protocols and procedures to be used by the IOUs to document and verify costs and benefits of major DSM program activities for shareholder earnings and measurement agreements for resource planning purposes.

13

“Protocols”

Gross Impacts Provides various types of models/approaches

acceptable and generally when they are applicable

Sample design requirements for approaches Reporting requirements Documentation requirements Measurement schedule

Net Impacts Retention Studies

14

“Protocols” Summary

Prescriptive protocols – Specify what, when and how

Contain ability to request a waiver from specified approaches and schedules; reviewed by CADMAC.

Supplied relative surety that if protocols were followed results would be acceptable.

15

ComparisonsEE Policy Manual

CA Evaluation Framework

“Protocols” IPMVP

Specific to energy efficiency programs Yes Yes Yes No

Protocols with known success parameters and avenue for disputes

No No Yes No

Energy impact evaluation procedures No Yes Yes Yes

Demand impact evaluation procedures No No No No

Program level sampling procedures No Yes Yes No

Net issues addressed No Yes Yes No

Specific reporting formats and procedures No No Yes No

Persistence impact evaluation procedures No No Yes No

Timing of evaluations discussed No Yes Yes No

16

Discussion PointsWhat are the relevant questions for our next set of protocols? Who are the ultimate users of the energy and demand impacts,

what data is required and when is the information needed? How can the protocols be structured such that you can put

resources where there is the most uncertainty? Do evaluations require precise demand impacts or can we use

the Savings Goals GWh to MW conversion? What are the ramifications of 3rd party programs being unable to

use billing analysis that includes nonparticipants? Will evaluation budgets support yet-to-be-determined precision

requirements for energy/demand impact evaluation as well as allowing for logic modeling and process evaluation?

Others?