SMALL BUSINESS ENERGY SAVINGS PROGRAM ......2016/02/13 · ComEd Small Business Energy Savings PY7...

© 2016 Navigant Consulting, Inc.

SMALL BUSINESS ENERGY SAVINGS

PROGRAM

FINAL

Energy Efficiency/Demand Response Plan:

Plan Year 7

(6/1/2014-5/31/2015)

Presented to

Commonwealth Edison Company

February 13, 2016

Prepared by:

Paul Higgins

Navigant Consulting, Inc.

Charles Ampong


Argene McDowell


www.navigant.com

Submitted to:

ComEd

Three Lincoln Centre

Oakbrook Terrace, IL 60181

Submitted by:


30 S. Wacker Drive, Suite 3100

Chicago, IL 60606

Contact:

Randy Gunn, Managing Director

312.583.5714

[email protected]

Jeff Erickson, Director

608.497.2322

[email protected]

Rob Neumann, Assoc. Dir.

312.583.2176

[email protected]

Acknowledgements

This report includes contributions from Rob Slowinski and Mary Thony, in addition to those individuals

listed above.

Disclaimer: This report was prepared by Navigant Consulting, Inc. (“Navigant”) for ComEd based upon

information provided by ComEd and from other sources. Use of this report by any other party for

whatever purpose should not, and does not, absolve such party from using due diligence in verifying the

report’s contents. Neither Navigant nor any of its subsidiaries or affiliates assumes any liability or duty of

care to such parties, and hereby disclaims any such liability.

mailto:[email protected]

mailto:[email protected]

ComEd Small Business Energy Savings PY7 Evaluation Report – Final Page iii

Table of Contents

E. Executive Summary .......................................................................................................... 1

E.1 Program Savings ................................................................................................................................. 2 E.2 Program Savings by Measure ............................................................................................................ 3 E.3 Impact Estimate Parameters for Future Use .................................................................................... 3 E.4 Program Volumetric Detail ................................................................................................................ 3 E.5 Results Summary................................................................................................................................. 4 E.6. Findings and Recommendations ....................................................................................................... 4

1 Introduction ....................................................................................................................... 7

1.1 Program Description ........................................................................................................................... 7 1.2 Evaluation Objectives ......................................................................................................................... 8

1.2.1 Impact Questions .................................................................................................................. 8 1.2.2 Process Questions ................................................................................................................. 8

2 Evaluation Approach ........................................................................................................ 9

2.1 Overview of Data Collection Activities ............................................................................................ 9 2.2 Verified Savings Parameters .............................................................................................................. 9

2.2.1 Verified Gross Program Savings Analysis Approach ...................................................... 9 2.2.2 Verified Net Program Savings Analysis Approach........................................................ 10

2.3 Process Evaluation ............................................................................................................................ 11

3 Gross Impact Evaluation ............................................................................................... 12

3.1 Tracking System Review .................................................................................................................. 12 3.2 Program Volumetric Findings ......................................................................................................... 14 3.3 Gross Program Impact Parameter Estimates ................................................................................. 18 3.4 Verified Gross Program Impact Results ......................................................................................... 20

4 Net Impact Evaluation ................................................................................................... 21

4.1 Verified Net Impacts ......................................................................................................................... 21 4.2 Evaluation Research Net-to-Gross Findings ................................................................................. 23

5 Findings and Recommendations ................................................................................. 24

6 Appendix .......................................................................................................................... 27

6.1 Evaluation Research Impact Approaches and Findings .............................................................. 27 6.1.1 Evaluation Research Gross Impact Parameter Estimates .............................................. 27

6.2 Detailed Process Findings ................................................................................................................ 30 6.2.1 Effectiveness of Program Implementation and Program Changes .............................. 30 6.2.2 Administration and Delivery ............................................................................................ 33 6.2.3 Effectiveness of Program Design and Processes ............................................................. 35

ComEd Small Business Energy Savings PY7 Evaluation Report – Final Page iv

6.3 PJM Data and Findings ..................................................................................................................... 37 6.4 Survey Instruments ........................................................................................................................... 39

6.4.1 Process Survey Instrument ................................................................................................ 39 6.4.2 Customer Net to Gross Survey Instrument ..................................................................... 51 6.4.3 Contractor In-Depth Interview Guide ............................................................................. 64

List of Figures and Tables

Figures

Figure 3-1. Share of SBES Ex Ante Energy Savings by End-use Type ............................................................. 17 Figure 3-2. Share of SBES Ex Ante Gross Energy Savings by Program Channel ........................................... 18 Figure 6-1. . First Became Aware of the SBES Program ..................................................................................... 30 Figure 6-2. Received Information or Heard About SBES Program .................................................................. 31 Figure 6-3. Customers Found Marketing Materials Useful .............................................................................. 32 Figure 6-4. Best Channels to Reach Small Business Customers ....................................................................... 33 Figure 6-5. Days From Project Approval to Equipment Installation – Lower and Upper Bounds ............. 34 Figure 6-6. Number of Days to Payment Receipt After Application Approved ............................................ 35

Tables

Table E-1. PY7 Total Program Electric Savings .................................................................................................... 2 Table E-2. PY7 Total IPA Program Electric Savings ............................................................................................. 2 Table E-3. PY7 Total EEPS Program Electric Savings .......................................................................................... 2 Table E-4. PY7 Program Results by Measure End-use ........................................................................................ 3 Table E-5. PY7 Volumetric Findings Detail ........................................................................................................... 3 Table E-6. PY7 Results Summary............................................................................................................................ 4 Table 2-1. Primary Data Collection Activities ....................................................................................................... 9

Table 2-2. Verified Savings Parameter Data Sources ......................................................................................... 10 Table 2-3. PY7 SBES NTG Research Results ........................................................................................................ 10 Table 3-1. PY7 Volumetric Findings Detail By Program Channel ................................................................... 15 Table 3-2. PY7 Volumetric Findings Detail for IPA and EEPS Programs ....................................................... 15 Table 3-3. PY7 Volumetric Findings Detail ......................................................................................................... 16 Table 3-4. PY7 SBES Ex-Ante and Verified Gross Savings Parameters ........................................................... 19 Table 3-5. PY7 Verified Gross Impact Savings Estimates by End-use ............................................................. 20 Table 4-1. PY7 Verified Net Impact Parameters ................................................................................................. 21 Table 4-2. PY7 Verified Net Impact Savings Estimates by End-use for all Projects ....................................... 22 Table 4-3. PY7 Verified Net Impact Savings Estimates For IPA and EEPS Programs................................... 22 Table 4-4. PY7 Net-to-Gross Research Findings from PY7................................................................................ 23 Table 6-1. Measure Level Gross Impact Results ................................................................................................. 27 Table 6-2. PY7 Verified Gross Impact Savings Estimates by Building Type .................................................. 29

ComEd Small Business Energy Savings PY7 Evaluation Report – Final Page 1

E. Executive Summary

This report presents a summary of the findings and results from the impact and process evaluation of the

program year seven (PY7) 1 Small Business Energy Savings (SBES) program, ComEd’s primary energy

efficiency program for small business customers. PY7 represents the program’s fourth full year of

operation.

The SBES program is designed to assist qualified ComEd non-residential customers2 to achieve electric

energy savings by educating them about energy efficiency (EE) opportunities through on-site

assessments conducted by specially-trained trade allies (TAs) and installation of no-cost direct-install (DI)

measures.3 Further savings are available to participating customers through incentives of 30 percent to 75

percent offered for select contractor-installed (CI) measures.

Notable program changes in PY7 included:

Additional measures: Multi-Family program common area measures moved from the Multi-

Family program to the SBES program, including LED lamps and fixtures, HP/RW T8s, LED exit

signs, new T8/T5 fixtures, occupancy sensors, and photocells.

Change in implementation contractor: Nexant, Inc. (Nexant) became the sole implementer for the

SBES program throughout ComEd’s service territory.4

Status as joint program: Starting in PY7 ComEd’s SBES program was no longer a joint program

with any of the gas companies operating in ComEd’s territory.5

The Small Business Program PY7 gross impact evaluation was based on engineering review and

verification of program measure savings using deemed input parameters and algorithms from the Illinois

Statewide Technical Reference Manual (TRM version 3.0).6 The PY7 net verified savings were calculated

based on net-to-gross ratio (NTGR) estimates from past evaluation research (PY5) and established by a

consensus process with the Illinois Statewide Advisory Group (SAG).7

1 The PY7 program year began June 1, 2014, and ended May 31, 2015. 2 To qualify, participants must be ComEd commercial or industrial customers with monthly peak demand levels no

greater than 100 kW. 3 No-cost direct-install measures include low-flow showerheads and faucet aerators, pre-rinse spray valves, vending

machine controls, cooling and vending misers, and compact fluorescent lamps (CFLs). 4 In PY4 and PY5 Nexant implemented the program for customers served by ComEd and Nicor Gas, while Franklin

Energy Services implemented the program for customers served by ComEd and Peoples Gas or North Shore Gas.

Nicor Gas de-linked its small business EE program from ComEd’s at the end of PY5. Peoples Gas and North Shore

Gas de-linked their small business EE programs from ComEd’s after PY6. 5 In PY4 and PY5 the program was joint between ComEd, Nicor Gas, Peoples Gas and North Shore Gas; in PY6 the

program was joint between ComEd, Peoples Gas and North Shore Gas. 6 State of Illinois Technical Reference Manual v3.0. Final as of June 24, 2014, effective June 1, 2014.

http://ilsagfiles.org/SAG_files/Technical Reference Manual 7 Source: ComEd_NTG_History_and_PY7_Recommendation_2014-02-28_Final_EMV_Recommendations.xlsx, which

is to be found on the IL SAG web site here: http://ilsag.info/net-to-gross-framework.html

http://ilsag.info/net-to-gross-framework.html


E.1 Program Savings

Table E-1 summarizes the electricity savings from the SBES Program. The program achieved verified net

energy savings of 173,705 megawatt-hours (MWh), verified net demand reduction of 33.74 megawatts

(MW) and verified net peak demand reduction of 29.37 MW.

Navigant calculated the SBES program’s verified net savings to be 173,705 MWh, its verified net demand

savings as 33.74 MW, and its verified net peak demand savings as 29.37 MW. These were allocated

between the EEPS and IPA portfolios as follows: IPA net energy savings of 100,001 MWh and net peak

demand savings of 16.22 MW. EEPS net energy savings of 73,704 MWh and net peak demand savings of

13.15 MW.

Table E-1. PY7 Total Program Electric Savings

Savings Category Energy Savings (MWh) Demand Savings (MW) Peak Demand Savings

(MW)

Ex Ante Gross Savings 182,959 35.82 31.22

Verified Gross Savings 182,847 35.51 30.91

Verified Net Savings 173,705 33.74 29.37

Source: ComEd tracking data and Navigant team analysis.

Table E-2 and Table E-3 summarize the allocation of PY7 SBES electricity savings between the Energy

Efficiency Portfolio Standard (EEPS) and Illinois Power Agency (IPA) portfolios.8

Table E-2. PY7 Total IPA Program Electric Savings


(MW)





Table E-3. PY7 Total EEPS Program Electric Savings


(MW)





8 ComEd allocated 100,000 net MWh to IPA based on the IPA budget, with the rest going to EEPS (ComEd PY7 Ex

Ante Savings.xlsx, 9-05-2015, and correspondence from ComEd program manager). Navigant identified 100,001 net

MWh for IPA and 73,704 net MWh for EEPS in the tracking data.


E.2 Program Savings by Measure

Table E-4 summarizes the PY7 SBES program savings by measure end-use category.

Table E-4. PY7 Program Results by Measure End-use

Measure Category

Ex Ante Gross

Savings (MWh)

Ex-Ante Gross

Peak Demand

Reduction (MW)

Verified Gross

Savings (MWh)

Verified Gross

Peak Demand

Reduction (MW)

Verified Gross MWh

Realization Rate

NTGR

Verified Net

Savings (MWh)

Verified Net Peak Demand

Reduction (MW)

Lighting 178,293 30.7 178,465 30.7 100% 0.95* 169,542 29.16

Refrigeration 4,618 0.49 4,325 0.18 94% 0.95* 4,109 0.18

Hot Water Efficiency 29 0.01 38 0.01 131% 0.95* 36 0.01

HVAC 19 0.02 19 0.02 100% 0.95* 18 0.02

Total 182,959 31.22 182,847 30.91 100% 0.95* 173,705 29.37

Source: ComEd tracking data and Navigant team analysis. * A deemed value. Source: ComEd_NTG_History_and_PY7_Recommendation_2014-02-28_Final_EMV_Recommendations.xlsx, which is available at http://ilsag.info/net-to-gross-framework.html.

E.3 Impact Estimate Parameters for Future Use

Navigant conducted NTG and spillover research in latter part of PY6 and early PY7 for future program

application. We estimated a combined participant free ridership and trade ally free ridership value of

0.11. Our research showed participant spillover was zero from the sample. However, we believe the

spillover estimate from our previous spillover study is still valid and will include that estimate (0.02) in

our draft recommendation for PY9 NTG values.

E.4 Program Volumetric Detail

As shown in the following Table E-5, the SBES program implemented 10,141 projects and installed

959,103 measures in PY7.

Table E-5. PY7 Volumetric Findings Detail

Participation Direct-Install Contractor-

Installed Total

Total Implemented Projects 205 10,120 10,141*

Total Participant Customers 203 9,792 9,800**

Total Program Measures9 422 958,681 959,103

Source: ComEd tracking data and Navigant team analysis. * Unique projects: excludes 184 duplicate projects which had both CI and DI measures installed. ** Unique customers: excludes 195 duplicate customer account numbers with both CI and DI measures installed.

9 For evaluation reporting purpose, if a lighting measure quantity is reported in the tracking system as connected

watts, watt reduced, or watts controlled, Navigant treated each row entry of such measure as one measure quantity

in this table. The actual connected watts, reduced watts, or watts controlled are reported in Section 3.2 at the

program-level analysis.


E.5 Results Summary

The following table summarizes the key metrics from PY7.

Table E-6. PY7 Results Summary

Units PY7

Net Energy Savings MWh 173,705

Net Summer Demand Reduction MW 33.74

Net Summer Coincident Peak Demand Reduction MW 29.37

Gross Savings MWh 182,847

Gross Summer Demand Reduction MW 35.51

Gross Summer Peak Demand Reduction MW 30.91

Program Realization Rate % 100%

Program NTG Ratio* # 0.95

Lighting Measures Installed # 941,994

Non-Lighting Measures Installed # 17,109

Participants (Tenant Units or Projects) # 10,141

Customers Touched (Property Accounts) # 9,800

Source: ComEd tracking data and Navigant team analysis. *A deemed value. Source: ComEd_NTG_History_and_PY7_Recommendation_2014-02-28_Final_EMV_Recommendations.xlsx, which is available at http://ilsag.info/net-to-gross-framework.html.

E.6. Findings and Recommendations

Navigant’s key findings and recommendations for the SBES program from the PY7 evaluation include:

Verified Net Impacts & NTGR

Finding 1. Navigant used deemed net-to-gross (NTG) ratio estimates from the Illinois SAG

consensus process to calculate net verified savings for both EEPS and IPA measures.10 The

overall net energy savings of 173,705 is 104 percent of the SBES PY7 net savings goal of

167,582 MWh. Navigant conducted customer and trade ally NTG research in PY7, the results

of which will form the basis for our recommended NTG ratio for prospective application in

PY9 (June 1, 2016 to May 31, 2017), to be submitted to the Illinois SAG in January 2016.

Verified Gross Impacts and Realization Rates

Finding 2. The PY7 SBES program achieved 182,847 MWh of verified gross energy savings and

30.91 MW of verified gross peak demand savings. The total verified gross energy savings is

112 MWh lower than the ex ante gross savings of 182,959 MWh, with an approximate 100

percent overall verified gross realization rate on energy savings.11 The program default

10 ComEd_NTG_History_and_PY7_Recommendation_2014-02-28_Final_EMV_Recommendations.xlsx, which may be

found on the IL SAG web site (http://ilsag.info/net-to-gross-framework.html). 11 Verified Gross Realization Rate (RR) = ratio of verified gross savings over ex ante gross savings.




lookup values and ex ante savings for most measures were consistent with the TRM (v3.0),

but some measures’ default savings values required further review. Most lighting measures

had verified gross realization rates of 100 percent. Some lighting measures required

adjustment to the correct TRM (v3.0) lighting interactive factors and hours of use values for

the installed space type reported in the tracking database. The evaluation adjustment for

bathroom and kitchen faucet aerators resulted in increased savings with a 194 percent

measure gross savings realization rate.

Recommendation 1. ComEd and Nexant should review the PY8 savings input assumptions for

refrigeration coolers and freezers with strip curtains to be consistent with the TRM (v4.0)

deemed values of 422 kWh savings per cooler and 2,978 kWh savings per freezer with strip

curtains. ComEd should track the actual quantity of refrigeration coolers and freezers with

strip curtains installed.

Recommendation 2. ComEd should ensure the lighting savings and input assumptions are

consistent with the building type reported in the tracking system. ComEd should update the

tracking system default input values for bathroom and kitchen aerators to reflect the

approved version of the TRM (v4.0) for PY8. The current ex ante savings per aerator ranges

from 70 to 88 kWh, and this should be updated to 137 kWh per faucet aerator.

Peak Demand Reduction

Finding 3. The program tracking database does not include peak demand and non-peak demand

savings. The evaluation team estimated peak (summer and winter) and non-peak demand

savings for each program measure and project using the applicable Illinois TRM (v3.0)

assumptions and research.

Program Participation

Finding 4. Navigant worked with ComEd and Nexant to identify and remove overlapping

projects and measure counts to avoid attributing savings to both the PY7 Small Business and

PY7 Business Instant Lighting Discounts (BILD) programs.12 The approach was based on

periodic tracking by ComEd of funds to both programs through IPA and EEPS. Navigant

reviewed the final projects, measure counts, and savings attributed to the SBES program.

Navigant verified that ComEd attributed 182,959 MWh as the ex ante savings for SBES after

the removal of the overlap savings, and 34,921 MWh were allocated to BILD as overlap

savings from SBES. The overlap comprised 204,119 lighting measure units, including Metal

Halides, Directional and Omnidirectional LEDs, LED Exit Signs, and Incandescent Bulbs

retrofit to LEDs or CFLs or Cold Cathode.

Recommendation 3. ComEd has proposed an attribution model for PY8 to identify and attribute

overlap savings to SBES or BILD based on program participation and the incentive

proportion paid by each program. Navigant’s initial assessment of the proposed model

suggests that the approach needs further review and refinement, considering the obvious

differences in program metrics such as building types or installed space, tracking measure

descriptions and incentive levels.

12 BILD is a midstream program that sells some of the equipment installed through the SBES Program. ComEd allows

SBES Trade Allies to buy eligible products from BILD distributors. ComEd allocated energy savings between SBES

and BILD for these common measures.


Recommendation 4. ComEd should consider standardizing and updating the measure

descriptions in the SBES tracking system to help streamline the overlap attribution process.

(For example, some measures in the SBES tracking system have bracketed numbers before

the measure ID – “[DI03] Incandescent 100W - 23W CFL,” versus “100W Incandescent to 23W

CFL”.) Several other measures may need standardized descriptions.

Process Findings

Finding 6. Some participants and trade allies expressed frustration at the spending cap, which

caused the SBES program manager to restrict program activities beginning in early January

2015. Some trade allies maintained that stopping funding midyear had hurt their businesses,

and the issue was also mentioned by some participants. Trade allies and customers make

plans assuming that advertised SBES program measures will be available.

Recommendation 6. ComEd should explore ways to bring the demand for subsidized EE

measures offered to its small C&I customers through the SBES program into better balance

with the available budget. We understand that ComEd has already taken an initial step

toward this goal in PY8, by implementing a pre-application step that must be completed and

submitted by the trade ally prior to undertaking any SBES-approved work through the

program. This should provide some much-needed insight into the program’s burn rate.


1 Introduction

1.1 Program Description

This report presents a summary of the findings and results from the impact and process evaluation of the

program year seven (PY7)13 Small Business Energy Savings (SBES) Program, ComEd’s primary energy

efficiency program for small business customers. PY7 represents the program’s fourth full year of

operation.

The SBES program is designed to assist qualified ComEd non-residential customers14 to achieve electric

energy savings by educating them about energy efficiency (EE) opportunities through on-site

assessments conducted by specially-trained trade allies (TAs) and installation of no-cost direct-install (DI)

measures.15 Further savings are available to participating customers through incentives of 30 percent to 75

percent offered for select contractor-installed (CI) measures.

Notable program changes in PY7 include:

Additional measures: Multi-Family program common area measures moved from the Multi-

Family program to the SBES program, including LED lamps and fixtures, HP/RW T8s, LED exit

signs, new T8/T5 fixtures, occupancy sensors, and photocells.

Change in implementation contractor: Nexant, Inc. (Nexant) became the sole implementer for the

SBES program throughout ComEd’s service territory.16

Status as joint program: Starting in PY7 ComEd’s SBES program was no longer a joint program

with any of the gas companies operating in ComEd’s territory.17

In PY8, ComEd plans to introduce a new customer assessment tool to be used by TAs, which will

provide new customers with more insight into potential energy savings opportunities. Also in

PY8, ComEd instituted a pre-application step to gain more real-time insight into the program’s

burn rate and also spur customers and TAs to give greater consideration to non-lighting

measures.

The evaluation team relied solely on the Illinois Statewide Technical Reference Manual (TRM)18 for

deemed gross savings verification for each program measure. The verified net impact evaluation

13 The PY7 program year began June 1, 2014, and ended May 31, 2015. 14 To qualify, participants must be ComEd commercial or industrial customers with monthly peak demand levels no

greater than 100 kW. 15 No-cost direct-install measures include low-flow showerheads and faucet aerators, pre-rinse spray valves, vending

machine controls, cooling and vending misers, and compact fluorescent lamps (CFLs). 16 In PY4 and PY5 Nexant implemented the program for customers served by ComEd and Nicor Gas, while Franklin

Energy Services implemented the program for customers served by ComEd and Peoples Gas or North Shore Gas.

Nicor Gas de-linked its small business EE program from ComEd’s at the end of PY5. Peoples Gas and North Shore

Gas de-linked their small business EE programs from ComEd’s after PY6. 17 In PY4 and PY5 the program was joint between ComEd, Nicor Gas, Peoples Gas and North Shore Gas; in PY6 the

program was joint between ComEd, Peoples Gas and North Shore Gas. 18 Illinois Statewide Technical Reference Manual for Energy Efficiency Version 3.0, available at:

http://www.ilsag.info/technical-reference-manual.html



approach applied the net-to-gross (NTG) ratios deemed through the Illinois State Advisory Group (IL

SAG) consensus process.19

1.2 Evaluation Objectives

The evaluation team identified the key researchable questions listed in the following sections.

1.2.1 Impact Questions

1. What are the program’s verified gross energy savings?

2. What are the program’s verified net energy savings?

3. What are the program’s verified demand savings?

4. What is the researched value for Net-to-Gross (NTG) ratio?

5. What updates are recommended for the Illinois Technical Reference Manual (TRM)?

1.2.2 Process Questions

1. What was the effectiveness of program implementation and program changes?

2. How effective was program administration and delivery?

3. How effective was the program design and processes?

4. How satisfied were participating customers and trade allies with the program?

5. What opportunities exist for program improvement?

19 ComEd_NTG_History_and_PY7_Recommendation_2014-02-28_Final_EMV_Recommendations.xlsx, found on the

IL SAG website: http://ilsag.info/net-to-gross-framework.html



2 Evaluation Approach

Navigant’s evaluation team reviewed the program tracking data and performed gross and net impact

calculations to inform verified energy and demand savings for PY7. The evaluation team evaluated the

gross savings by (1) reviewing the tracking system, (2) comparing the use of measure algorithms in the

tracking database to their use in the Illinois TRM v3.0 to ensure that they are appropriately applied and

(3) cross-checked totals.

2.1 Overview of Data Collection Activities

The core data collection activities included review of the program’s tracking data and verification of

measures savings against the Illinois TRM v3.0. The full set of data collection activities is shown in the

following table.

Table 2-1. Primary Data Collection Activities

What Who Target

Completes PY7

When Comments

Review Program Tracking Database

Participants All Jun - Oct

2015 Source of information for verified gross analysis

Review Program Measures in TRM

Illinois Statewide Technical Reference Manual for Energy Efficiency Version 3.0

All Jun - Oct

2015

Source of information for verified gross analysis

Program Material Review

Program Documents All Apr - Aug

2015 Source of information for process research

NTG Telephone Survey

Participating Customers 70 Jan - Jun

2015 Source of information for NTG research

NTG Telephone Interviews

Trade Allies 12 Aug - Oct

2015 Source of information for NTG research

Process Telephone Survey

Participating Customers 102 Jun 2014 - Mar 2015

Source of information for process research

In Depth Interviews Program Management 2 Mar - May

2015

Includes interviews with staff from ComEd and Nexant

2.2 Verified Savings Parameters

2.2.1 Verified Gross Program Savings Analysis Approach

Navigant estimated verified unit savings for each program measure using deemed impact algorithm

sources found in the Illinois TRM v3.0. Verified unit savings values reflect evaluation adjustments to ex-

ante unit savings values based on Navigant’s measure review. The tracking data for the PY7 SBES

program evaluation came from ComEd’s program tracking system, uploaded on the ComEd SharePoint

site for evaluators, and extracted by Navigant on October 7, 2015. Navigant reviewed the program

tracking system and procedures to verify that the program accurately reported measure counts. Navigant

engaged with ComEd and the IC to resolve the issue with overlap of savings and projects/measures

installed through the SBES program and the BILD program. The SBES program verified gross savings


were determined as the product of verified unit savings values (energy and demand savings) and verified

measure quantities.

2.2.2 Verified Net Program Savings Analysis Approach

Verified net energy and demand (coincident peak and overall) savings were calculated by multiplying

the verified gross savings estimates by a net-to-gross ratio (NTGR). In PY7, the NTGR estimates used to

calculate the net verified savings were based on past evaluation research and deemed through the Illinois

Stakeholder Advisory Group (IL SAG) consensus process.

Table 2-2 presents the key parameters and the references used in the verified gross and net savings

calculations.

Table 2-2. Verified Savings Parameter Data Sources

Verified Gross and Net Input Parameter Value Data Source Deemed or Evaluated

NTGR – all program measures 0.95 IL SAG Spreadsheet† Deemed

Measure Quantity Installed Vary Program Tracking System Evaluated

Verified Energy Gross Realization Rate 100% Program Tracking Data Review Evaluated

Verified Peak Demand Gross Realization Rate 100% Program Tracking Data Review Evaluated

All lighting measures delta watts Vary Illinois TRM v3.0, Section 4.5.1 Deemed

Lighting In-Service Rate 1.00 Illinois TRM v3.0, Section 4.5.1 Deemed

DI Showerhead In-Service Rate 0.98 Illinois TRM v3.0, Section 4.3.3 Deemed

DI Bathroom & Kitchen Aerator In-Service Rate 0.95 Illinois TRM v3.0, Section 4.3.2 Deemed

HVAC & Refrigeration Measures Inputs Vary Illinois TRM v3.0, Sections 4.6 Deemed

Source: Navigant analysis of ComEd tracking data (10-07-2015 data extract). Note: Gross realization rate is 100% for all measures’ peak and non-peak demand savings. † Source: ComEd_NTG_History_and_PY7_Recommendation_2014-02-28_Final_EMV_Recommendations.xlsx, which is available at http://ilsag.info/net-to-gross-framework.html

Navigant conducted NTG research in PY7 through a participant NTG survey and in-depth interviews

with TAs, for the purpose of updating the recommended NTGR. Key results are shown in Table 2-3.

Additional details are presented in section 4.2.

Table 2-3. PY7 SBES NTG Research Results

Data Collection Method Sampling Frame Number of Completes Free-Ridership Score

NTG Participant Survey 6,209 Projects 70 16%

Trade Ally Interviews 74 Trade Allies 12 5%

Source: Navigant analysis of primary research data.



2.3 Process Evaluation

PY7 process evaluation activities involved information gathering on effectiveness of the current program

design, administration, delivery, implementation processes, customer and program partner experience

and satisfaction, and opportunities for program improvement. The process analysis included a synthesis

of both qualitative and quantitative data collected during the program implementer and program

coordinator interviews, the end-user customer surveys and the trade ally surveys or in-depth interviews.

Navigant conducted participant process surveys on a rolling basis beginning with the latter part of PY6

and continuing through the first half of PY720, and also conducted in-depth interviews with trade allies in

the latter half of PY7. The decision to conduct the participant surveys in small batches over an extended

period of time21 was part of an effort, undertaken at ComEd’s suggestion, to move toward a more “real-

time” evaluation of the SBES program, to allow the evaluation team to alert program managers if any

problems were encountered that could be addressed before the end of the program year.22 Navigant

conducted NTG research through a separate customer survey and in-depth interviews with TAs for the

purpose of updating the recommended NTGR value for future use.

20 The process survey instrument is included as an attachment in the Appendix. 21 Previously, Navigant has conducted similar participant process surveys in batch mode. 22 Navigant did encounter one complaint from a program participant in PY7 involving a dispute with a trade ally that

was passed along to SBES program managers. We understand that this led to ComEd and the implementer taking

corrective action.


3 Gross Impact Evaluation

The PY7 SBES program achieved 182,847 MWh of verified gross savings, 35.51 MW of verified gross

demand savings, and 30.91 MW of verified gross peak demand savings. The program’s verified gross

realization rate was 100 percent for energy and 100 percent for demand savings. The high realization rate

on energy and demand savings was due mainly to only minor needs for evaluation adjustments to some

input assumptions used to calculate ex-ante measure savings.

3.1 Tracking System Review

Navigant worked with ComEd and Nexant during the course of PY7 to review the SBES program and the

Standard program brochures of default lighting fixture wattages used to assist customers and trade allies

with wattage reduction calculations. Navigant identified discrepancies between the fixture default

wattage tables in the two brochures. For the fixture types with discrepancies, Navigant determined

whether the fixture wattage was identified within the Illinois TRM. Additional sources were also

reviewed to further verify the fixture wattage as needed. Details of the findings presented to ComEd and

Nexant are provided in Appendix 6.1.1.

Navigant also worked with ComEd and Nexant to identify and remove overlapping projects and

measure counts to avoid attributing savings to both the PY7 Small Business and PY7 Business Instant

Lighting Discounts (BILD) programs.23 The approach was based on periodic tracking by ComEd of funds

to both programs through IPA and EEPS. Navigant reviewed the final projects, measure counts, and

savings attributed to the SBES program, as reported and downloaded from ComEd evaluation SharePoint

site on October 7, 2015. Navigant verified that ComEd attributed 182,959 MWh as the ex ante savings for

SBES after the removal of the overlap savings, and 34,921 MWh were allocated to BILD as overlap

savings from SBES. The overlap comprised 204,119 lighting measure units, including Metal Halides,

Directional and Omnidirectional LEDs, LED Exit Signs, and Incandescent Bulbs retrofit to LEDs or CFLs

or Cold Cathode.

Navigant reviewed the tracking data to verify the completeness and accuracy of the tracking system

default measure savings inputs, using the TRM (v3.0), ComEd’s SBES program default measure lookup

savings spreadsheet24, and supporting ComEd work papers where relevant25, to verify input assumptions

for other deemed or non-deemed measures. Navigant verified that the program ex ante savings for most

measures were consistent with the TRM and ComEd’s work paper and default lookup tables, with

exceptions noted below that need an update of the default savings input assumptions.

Key findings from the tracking system review include the following:



and BILD for these common measures. 24 PY7 Measure Savings by Facility_w_ComEd IDs - v4.xlsx 25 PY7 ComEd Measure Work papers 3-25-2015.pdf


1. The tracking default savings for most refrigeration measures were consistent with the TRM (v3.0)

and ComEd’s work paper and default lookup tables. The tracking measure count and ex ante

savings for refrigeration coolers and freezers with strip curtains installed deserve further review.

Navigant inquired about the assumptions built into the measure savings calculation and the

tracking measure quantity. Upon additional data submitted by ComEd, Navigant verified that ex

ante energy savings reported in the tracking system have been revised over the course of the

program year. Within some periods of the program year, Nexant calculated savings on a per-unit

(cooler or freezer) basis, but in other periods Nexant calculated savings by linear feet or by square

feet of refrigeration cooler or freezer with strip curtains installed.26 The former approach is

consistent with the TRM (v3.0) which actually deemed the measure savings by unit cooler or

freezer with curtains installed.

Due to lack of uniform methodology to classify the unit of measure quantity and savings to allow

the use of the TRM deemed values, Navigant determined that this measure should be treated as a

custom measure for PY7. In this regard, Navigant reviewed the custom savings input

assumptions in the tracking system. The evaluation verified total gross savings for coolers and

freezer with strip curtains were 243 MWh rather than 536 MWh total ex ante savings (a reduction

of 293 MWh which is 45 percent realization rate for the measure). The reduction of the measure

claimed savings by 293 MWh was consistent with ComEd’s estimate after Navigant brought the

savings discrepancy to its attention. ComEd and Nexant should revise the PY8 savings input

assumptions for refrigeration coolers and freezers with strip curtains to be based on the TRM

(v4.0) deemed values of 422 kWh savings per cooler or 2,978 kWh savings per freezer with strip

curtains installed. Nexant should track the actual quantity of refrigeration coolers and freezers

with strip curtains installed. Additional fields can be added to the tracking system to track the

total linear feet or square feet of strip curtains installed.

2. The tracking system default savings for bathroom and kitchen faucet aerators have not been

updated from the PY6 ex ante values, which had an errata and were corrected by Navigant using

the TRM (v3.0) in reported in the PY6 evaluation report. The PY7 tracking default savings ranges

from 70 kWh to 88 kWh per faucet aerator. Navigant calculated 137.48 kWh as the savings per

faucet aerator, assuming an average of 5,000 annual gallons of mixed water (mixture of hot water

from water heater line and cold water line) per faucet. The verified gross savings from faucet

aerators were 9 MWh more compared to the ex ante savings, with a 194 percent verified gross

realization rate.

3. The tracking system default savings and delta watts assumptions for lighting measures were

consistent with the TRM and ComEd’s SBES program default measure lookup savings

spreadsheet. Most of the lighting measures had a verified gross realization rate of 100 percent,

but there were a few projects with lighting savings that Navigant changed, based on the reported

26 Navigant followed up with ComEd on the measure discrepancy, and feedback from ComEd on December 3, 2015

indicated that the ex ante savings were calculated as 120.8 kWh per linear foot of cooler and 848.0 kWh per linear foot of

freezer (for curtains installed between June 1, 2015 to March 31, 2015). The ex ante savings were calculated as 15.1

kWh per square foot of cooler and 106.0 per square foot of freezer (for curtains installed between April 1, 2015 to May 31,

2015). The lack of uniform approach for calculating savings could be resolved if ComEd tracked all savings by unit

cooler or freezer with strip curtains as deemed by the TRM (v3.0).


building type and the TRM interactive factors and hours of use for lighting measures. The

lighting verified gross savings was 172 MWh more than the ex ante savings.

4. Navigant found some measures with negative quantities and savings which in some cases

aggregated to make some projects have negative overall savings. Previous conversation with

Nexant revealed that “when errors or adjustments are found after a project has already been

processed, Nexant does not go back and change the project in the tracking system, but [instead]

creates a subproject with the corrections.”27 Navigant did not adjust the reported measure

quantities based on this observation.

5. Navigant observed that several of the program measures have multiple descriptions in the

tracking system. ComEd should consider standardizing and updating tracking measure

descriptions from the program implementer (ComEd can use either “[L47] Occupancy Sensor” or

“Occupancy Sensors” but not both. This change may apply to several other measures with

multiple descriptions).

6. Navigant verified that ComEd attributed 182,959 MWh as the ex ante savings for SBES program,

and 34,921 MWh were allocated to BILD as overlap savings from SBES program. The overlap

comprised of 204,119 unit quantities of lighting measures including Metal Halides, Directional

and Omnidirectional LEDs, LED Exit Signs, and Incandescent Bulbs retrofit to LEDs or CFLs or

Cold Cathode.

7. ComEd moved the Multi-Family program common area measures to other Smart Ideas for Your

Business portfolio programs, including the SBES program. These measures included LED lamps

and fixtures, HP/RW T8s, LED exit signs, new T8/T5 fixtures, occupancy sensors, and photocells.

Navigant found the tracked ex ante savings from multi-family common area measures summed

to 5,582 MWh. Of this, ComEd allocated 1,639 MWh to the SBES program and the remaining

3,943 MWh to the BILD program as overlap savings.

3.2 Program Volumetric Findings

Table 3-1 shows the program volumetric findings disaggregated by program delivery channel. The SBES

program in PY7 implemented 10,141 projects and 959,103 measures, with 9,800 participants.

27 Email correspondence with the Program Manager at Nexant on 10-30-2014.


Table 3-1. PY7 Volumetric Findings Detail By Program Channel

Participation Direct-Install Contractor-Installed Total

Total Implemented Projects 205 10,120 10,141*

Total Participant Customers 203 9,792 9,800**

Total Program Measures28 422 958,681 9,59,103

Source: ComEd tracking data and Navigant team analysis. * Unique projects: excludes 184 duplicate projects which had both CI and DI measures installed. ** Unique customers: excludes 195 duplicate customer account numbers with both CI and DI measures installed.

Table 3-2 shows the program volumetric findings disaggregated for the IPA and EEPS program

categories. The IPA program comprised 2,105 projects with 454,677 contractor installed measures. The

EEPS program comprised 8,036 projects with 504,426 measures (422 direct install measures and 504,004

contractor installed measures).

Table 3-2. PY7 Volumetric Findings Detail for IPA and EEPS Programs

Participation IPA EEPS Total

Total Implemented Projects 2,105 8,036 10,141

Total Participant Customers 2,076 7,724 9,800

Total Program Measures 454,677 504,426 9,59,103


28 For evaluation purposes, if lighting measure quantity is reported in the tracking system as connected watts, watt

reduced or watts controlled, each row entry of such measure is treated as one measure quantity in this table. The

actual connected watts, reduced watts or watts controlled are reported in Table 3-3.


Table 3-3 below provides additional measure details for the direct install and the contractor installed

measures.

Table 3-3. PY7 Volumetric Findings Detail

Measure Unit Install Type Ex Ante Measure

Count Verified Measure

Count

Bath & Kitchen Aerators Each Direct Install 141 141

Vending Miser Each Direct Install 36 36

Showerheads Each Direct Install 21 21

Pre Rinse Sprayers Each Direct Install 7 7

Cooler Miser Each Direct Install 17 17

Beverage Machine Controls Each Direct Install 49 49

Reach-in (Novelty) Cooler Controls Each Direct Install 150 150

Snack Machine Controls Each Direct Install 1 1

T12 to HPT8/LW Retrofit Fixture Contractor Install 266,535 266,535

LED Exit Sign Sign Contractor Install 5,195 5,195

Exterior LED Each Contractor Install 399 399

Delamping T12 w/wo Reflector to HPT8 Fixture Contractor Install 45,036 45,036

4, 6-Lamp HPT8 w/HBF High-bay Fixture Contractor Install 5,031 5,031

Occupancy Sensors Watt Controlled Contractor Install 463,943 463,943

GREM – PTAC Each Contractor Install 17 17

Night Covers/Curtains Each Contractor Install 6 6

Standard T8 to HPT8/RW Lamp Contractor Install 454,109 454,109

2,3-Foot T8 Lamp and Ballast Fixture Contractor Install 13,819 13,819

Removed 2,4,8-Foot Lamp w/wo Reflector Fixture Contractor Install 148,288 148,288

LED Fixtures Watt Reduced Contractor Install 3,845,006 3,845,006

New T8/T5 Fixtures with Electronic Ballasts Watt Reduced Contractor Install 4,581,111 4,581,111

LED Refrigerated Display Case Lighting Open/Closed

Door Contractor Install 793 793

Photocells Watt Controlled Contractor Install 47,984 47,984

Outdoor LED Channel Sign ≤ 2 Feet Letter Contractor Install 36 36

Induction Fixtures Watt Reduced Contractor Install 5,687 5,687

Daylighting Controls Watt Controlled Contractor Install 2,184 2,184

EC Motor for Walk-in Cooler or Freezer Motor Contractor Install 1,137 1,137

Anti-Sweat Heater Controls for Glass Door Cooler or Refrigerator

Linear Foot Contractor Install 9,131 9,131

Strip Curtains for Cooler/Freezer Sq. Ft, or Ln Ft of Door

Contractor Install Sf= 4,085,

Lf= 941 Sf= 4,085,

Lf= 941

Auto Closer for Walk-in Cooler/Freezer Each Contractor Install 301 301

Evaporator Fan Controls on EC Motor Motor Contractor Install 1,069 1,069

Time Clocks for Lighting Connected Watts Contractor Install 2,349 2,349

Total 9,904,614 9,904,614



Figure 3-1 disaggregates the gross savings type. Overall, lighting measures contributed 98 percent of the

ex ante gross savings in PY7, and non-lighting measures (including hot water efficiency measures,

refrigeration measures and GREM-PTAC) contributed the remaining 2 percent.

Figure 3-1. Share of SBES Ex Ante Energy Savings by End-use Type

Source: Navigant analysis of ComEd tracking data

Figure 3-2 disaggregates the total program ex ante gross savings into its IPA and EEPS program shares.

The IPA and EEPS shares were 56 percent and 44 percent, respectively.


Figure 3-2. Share of SBES Ex Ante Gross Energy Savings by Program Channel


3.3 Gross Program Impact Parameter Estimates

Navigant estimated verified per-unit savings for each program measure using impact algorithm sources

found in the TRM for deemed measures29, and using evaluation research for non-deemed measures.

Navigant used ComEd’s SBES program default measure lookup savings spreadsheet30 with the

supporting ComEd work papers31 to verify input assumptions for other deemed or non-deemed

measures. Table 3-4 presents the key parameters and the references used in the verified gross and net

savings calculations.

29 Source: http://www.ilsag.info/technical-reference-manual.html 30 PY7 Measure Savings by Facility_w_ComEd IDs - v4.xlsx 31 PY7 ComEd Measure Work papers 3-25-2015.pdf


Table 3-4. PY7 SBES Ex-Ante and Verified Gross Savings Parameters

Measure Ex-Ante Gross Savings

(kWh/unit) Verified Gross Savings

(kWh/unit) Method* Source

Beverage Machine and Snack Machine Controls

1613 343

1613 343

Deemed IL TRM v3.0, Section 4.5.8

Cooler Miser, Reach-in (Novelty) Cooler Controls

1,210 1,210 Deemed IL TRM v3.0, Section 4.5.1

Showerheads 436.1 436.1 Deemed IL TRM v3.0, Section 4.5.2

Pre-rinse Sprayer 1,385 1,385 Deemed IL TRM v3.0, Section 4.5.2

Vending Miser 1,613 1,613 Deemed IL TRM v3.0, Section 4.5.4

Bath & Kitchen Aerators Vary from 70 kWh to 88

kWh 137.48 Deemed

IL TRM v3.0, Section 4.3.2

HPT8/LW Retrofit

Vary Vary. Adjusted based on verified

delta watts or building type interactive effects

Deemed IL TRM v3.0, Section 4.5

Delamping T12 to HPT8/RWT8

LED Lamps and Fixtures

Occupancy Sensors & Daylighting

Exterior LED

HID

4, 6-Lamp HPT8 w/HBF High-bay

Delamping & Removed Watts w/o reflector

LED Exit Sign

Time Clock & Photocell Vary Verified as acceptable Evaluated ComEd work papers

EC Motor, Walk-in & Reach-in

411 or 392 for walk-in, and 345 for reach in

401.04 Deemed IL TRM v3.0, Section 4.6.4

Strip Curtains for Coolers Strip Curtain for Freezers

Based on linear foot or square foot with custom

input Verified with adjustments Evaluated

ComEd work papers

Other Refrigeration Measures

Vary Verified as acceptable with

comments Deemed

TRM./ComEd workpapers

GREM - PTAC 1,117 1,117 Evaluated TRM/ComEd work papers

Source: Navigant analysis of ComEd tracking data * Deemed values are from Illinois TRM v3.0, available at http://www.ilsag.info/technical-reference-manual.html.



3.4 Verified Gross Program Impact Results

The ComEd PY7 SBES program reported ex-ante gross energy savings of 182,959 MWh. Evaluation

adjustments described in the previous sections resulted in evaluation verified gross energy savings of

182,847 MWh, verified gross peak demand savings of 30.91 MW. The program achieved 100 percent gross

realization rate on electricity savings. Table 3-5 presents the details of the verified savings by end-use

category and by program delivery channel.

Savings from DI measures contributed 378 MWh (0.2 percent) of the Program’s PY7 verified gross

savings. The CI measures contributed 185,068 MWh (99.8 percent) of PY7 verified gross savings. Verified

gross savings from all lighting measures were 178,465 MWh (98 percent) of the PY7 gross savings, and

verified gross savings from non-lighting measures were 4,382 MWh (2 percent). Details of the measure

level verified gross savings and realization rates are provided in the Appendix 6.1.

Table 3-5. PY7 Verified Gross Impact Savings Estimates by End-use

Program Channel Sample (90/10 Significance*)

Energy Savings (MWh) Coincident Peak Demand

Savings

Direct Install

Contractor Install

Direct Install

Contractor Install

Lighting

Ex Ante Gross Savings

NA†

- 178,293 - 30.70

Verified Gross Realization Rate**

- 100% - 100%

Verified Gross Savings - 178,465 - 30.70

Lighting Sub-total 178,465 30.70

Non-Lighting


NA†

368 4,298 0.01 0.52


103% 93% 100% 100%

Verified Gross Savings 378 4,004 0.01 0.20

Non-lighting Sub-total 4,382 0.21

Program Total Savings


NA†

182,959 31.22


100% 100%

Verified Gross Savings 182,847 30.91

Source: Navigant analysis of ComEd tracking data NA† indicates that the Illinois TRM v3.0 determines the gross savings. ** Based on evaluation research findings


4 Net Impact Evaluation

4.1 Verified Net Impacts

Navigant calculated the SBES program’s verified net savings to be 173,705 MWh, its verified net demand

savings as 33.74 MW, and its verified net peak demand savings as 29.37 MW. These were allocated

between the EEPS and IPA portfolios as follows: IPA net energy savings of 100,001 MWh and net peak

demand savings of 16.22 MW. EEPS net energy savings of 73,704 MWh and net peak demand savings of

13.15 MW.

Based on the Illinois Stakeholder Advisory Group (IL SAG) consensus process, NTG values for this

program are deemed prospectively and used to calculate verified net savings.32 Table 4-1 shows deemed

NTG values from the IL SAG consensus process.

Table 4-1. PY7 Verified Net Impact Parameters

End-use NTGR Source

Lighting 0.95 IL SAG

Non-lighting 0.95 IL SAG

Source: “ComEd_NTG_History_and_PY7_Recommendation_2014-02-28_Final_EMV_Recommendations.xlsx,”

available on the IL SAG website: http://ilsag.info/net-to-gross-framework.html

Table 4-2 presents the program net savings at the measure end-use level and by program channel. As

indicated, the overall savings from DI measures was 359 MWh (0.2 percent) of the total SBES program net

energy savings in PY7, whereas CI measures accounted for 173,346 MWh (99.8 percent) of the PY7 net

savings. Net savings from all lighting measures accounted for 169,542 MWh (96 percent) of PY7 Program

net savings, while net savings from non-lighting measures amounted to 6,631 MWh (4 percent).

The overall net energy savings of 173,705 is 104 percent of the SBES PY7 net savings goal of 167,582 MWh.

32 Source: ComEd_NTG_History_and_PY7_Recommendation_2014-02-28_Final_EMV_Recommendations.xlsx, found

on the IL SAG website: http://ilsag.info/net-to-gross-framework.html



Table 4-2. PY7 Verified Net Impact Savings Estimates by End-use for all Projects

Program Channel

Energy Savings (MWh) Coincident Peak Demand Savings

Direct Install

Contractor Install

Direct Install

Contractor Install

Lighting

Ex Ante Gross Savings - 178,293 - 30.70


- 100%

- 100%

Verified Gross Savings - 178,465 - 30.70

NTG Ratio* 0.95 0.95 0.95 0.95

Verified Net Savings 169,542 29.18 Non-Lighting Ex Ante Gross Savings 368 4,298 0.01 0.52


103% 93%

100% 100%

Verified Gross Savings 378 4,004 0.01 0.20

NTG Ratio* 0.95 0.95 0.95 0.95

Verified Net Savings 359 3,804 0.01 0.19 Program Total Savings Ex Ante Gross Savings 182,959 31.22


100%

100%

Verified Gross Savings 182,847 30.91

NTG Ratio* 0.95 0.95

Verified Net Savings 359 173,705 29.37

Source: Navigant analysis of ComEd tracking data *A deemed value from the IL SAG consensus process “ComEd_NTG_History_and_PY7_Recommendation_2014-02-28_Final_EMV_Recommendations.xlsx,” available on the IL SAG website: http://ilsag.info/net-to-gross-framework.html † NA indicates that the Illinois TRM v3.0 was used to determine gross savings.

Table 4-3 summarizes the allocation of total electricity savings between the EEPS and IPA portfolios.33

The IPA category realized net energy savings of 100,001 MWh and net peak demand savings of 16.22

MW. The EEPS category realized net energy savings of 73,704MWh and net peak demand savings of

13.15 MW.

Table 4-3. PY7 Verified Net Impact Savings Estimates For IPA and EEPS Programs

Savings Category EEPS IPA Total

Verified Net Savings (MWh) 73,704 100,001 173,705

Verified Net Demand Savings (MW) 14.43 19.31 33.74

Verified Net Peak Demand Savings (MW) 13.15 16.22 29.37


33 ComEd allocated 100,000 net MWh to IPA based on the IPA budget, with the rest going to EEPS (ComEd PY7 Ex

Ante Savings.xlsx, 9-05-2015, and correspondence from ComEd program manager). Navigant identified 100,001 net

MWh for IPA and 76,156 net MWh for EEPS in the tracking data.



4.2 Evaluation Research Net-to-Gross Findings

Navigant conducted net-to-gross research in PY7 using a participant customer survey and in-depth

interviews with participating trade allies. The sampling frame consisted of 6,209 completed SBES projects

obtained from an extract from the program tracking database performed in December 2014. After

removing duplicate and unusable records, Navigant randomly selected a sample of 847 projects for

study, from which a total of 70 finished the survey. The survey was conducted early in 2015.34 Based on

the survey results, Navigant calculated a free-ridership score of 16 percent.

Navigant also conducted net-to-gross research using in-depth interviews with twelve participating SBES

TAs drawn from a list of 74 provided by the program implementer. Navigant merged the TA list with a

project-level extract from the program tracking database containing project ex ante savings, sorted the

TAs by the total ex ante energy savings their projects represented, and on that basis divided them into

three groups: small, medium, and large. Navigant staff interviewed four TAs from each group to obtain a

wide range of TA experiences and opinions. The interviewed TAs represented approximately 30 percent

of the total ex ante energy savings in the extract. The interviews were conducted from August to October

2015.35 Based on the results of the TA interviews, Navigant calculated a free-ridership score of 5 percent.

These data, summarized in Table 4-4, will be used to update the recommended NTGR value for

prospective use beginning in PY9.36

Table 4-4. PY7 Net-to-Gross Research Findings from PY7

Data Collection Method Sampling Frame Number of Completes Free-Ridership Score

NTG Participant Survey 6,209 Projects 70 16%

Trade Ally Interviews 74 Trade Allies 12 5%

Source: Navigant analysis of primary research data.

We estimated a combined participant free ridership and trade ally free ridership value of 0.11. Our

research showed participant spillover was zero from the sample. However, we believe the spillover

estimate from our previous spillover study is still valid and will include that estimate (0.02) in our draft

recommendation for PY9 NTG values.

34 The survey instrument is included as an attachment in the Appendix. 35 The interview guide is included as an attachment in the Appendix. 36 The recommended prospective NTGR will be delivered in a separate memorandum in January 2016.


5 Findings and Recommendations

This section summarizes the key impact and process findings and recommendations.

Verified Net Impacts & NTGR

Finding 1. Navigant used deemed net-to-gross (NTG) ratio estimates from the Illinois SAG

consensus process to calculate net verified savings for both EEPS and IPA measures.37 The

overall net energy savings of 173,705 is 104 percent of the SBES PY7 net savings goal of

167,582 MWh. Navigant conducted customer and trade ally NTG research in PY7, the results

of which will form the basis for our recommended NTG ratio for prospective application in

PY9 (June 1, 2016 to May 31, 2017), to be submitted to the Illinois SAG in January 2016.

Verified Gross Impacts and Realization Rates

Finding 2. The PY7 SBES program achieved 182,847 MWh of verified gross energy savings and

30.91 MW of verified gross peak demand savings. The total verified gross energy savings is

112 MWh lower than the ex ante gross savings of 182,959 MWh, with an approximate 100

percent overall verified gross realization rate on energy savings.38 The program default

lookup values and ex ante savings for most measures were consistent with the TRM (v3.0),

but some measures’ default savings values required further review. Most lighting measures

had verified gross realization rates of 100 percent. Some lighting measures required

adjustment to the correct TRM (v3.0) lighting interactive factors and hours of use values for

the installed space type reported in the tracking database. The evaluation adjustment for

bathroom and kitchen faucet aerators resulted in increased savings with a 194 percent

measure gross savings realization rate. Navigant determined that savings from refrigeration

coolers and freezers with strip curtains should be treated as custom savings in PY7 because

the ex ante savings were tracked in kWh per linear feet or square feet rather than per unit

cooler or freezer with strip curtains as deemed in the TRM (v3.0). Navigant reviewed the

savings discrepancy with ComEd and determined the measure ex ante savings should be

reduced by 293 MWh, which produced 45 percent gross realization rate for the measure.

Recommendation 1. ComEd and Nexant should review the PY8 savings input assumptions for

refrigeration coolers and freezers with strip curtains to be consistent with the TRM (v4.0)

deemed values of 422 kWh savings per cooler and 2,978 kWh savings per freezer with strip

curtains. ComEd should track the actual quantity of refrigeration coolers and freezers with

strip curtains installed. Additional fields can be added to the tracking system to track the

total linear feet or square feet of strip curtains installed.

Recommendation 2. ComEd should ensure the lighting savings and input assumptions are

consistent with the building type reported in the tracking system. ComEd should update the

tracking system default input values for bathroom and kitchen aerators to reflect the

approved version of the TRM (v4.0) for PY8. The current ex ante savings per aerator ranges

from 70 to 88 kWh, and this should be updated to 137 kWh per faucet aerator.

37 ComEd_NTG_History_and_PY7_Recommendation_2014-02-28_Final_EMV_Recommendations.xlsx, which may be

found on the IL SAG web site (http://ilsag.info/net-to-gross-framework.html). 38 Verified Gross Realization Rate (RR) = ratio of verified gross savings over ex ante gross savings.



Peak Demand Reduction

Finding 3. The program tracking database does not include peak demand and non-peak demand

savings. The evaluation team estimated peak (summer and winter) and non-peak demand

savings for each program measure and project using the applicable Illinois TRM (v3.0)

assumptions and research.

Program Participation

Finding 4. Navigant worked with ComEd and Nexant to identify and remove overlapping

projects and measure counts to avoid attributing savings to both the PY7 Small Business and

PY7 Business Instant Lighting Discounts (BILD) programs.39 The approach was based on

periodic tracking by ComEd of funds to both programs through IPA and EEPS. Navigant

reviewed the final projects, measure counts, and savings attributed to the SBES program.

Navigant verified that ComEd attributed 182,959 MWh as the ex ante savings for SBES after

the removal of the overlap savings, and 34,921 MWh were allocated to BILD as overlap

savings from SBES. The overlap comprised 204,119 lighting measure units, including Metal

Halides, Directional and Omnidirectional LEDs, LED Exit Signs, and Incandescent Bulbs

retrofit to LEDs or CFLs or Cold Cathode.

Recommendation 3. ComEd has proposed an attribution model for PY8 to identify and attribute

overlap savings to SBES or BILD based on program participation and the incentive

proportion paid by each program. Navigant’s initial assessment of the proposed model

suggests that the approach needs further review and refinement, considering the obvious

differences in program metrics such as building types or installed space, tracking measure

description and incentive levels. Navigant recommends that ComEd simplify the attribution

model and use for instance, a simple weighting scheme based on the identified measure type

and counts.

Recommendation 4. ComEd should consider standardizing and updating the measure

descriptions in the SBES tracking system to help streamline the overlap attribution process.

(For example, some measures in the SBES tracking system have bracketed numbers before

the measure ID – “[DI03] Incandescent 100W - 23W CFL,” versus “100W Incandescent to 23W

CFL”.) Several other measures may need standardized descriptions.

Finding 5. ComEd moved the Multi-Family program common area measures to other Smart Ideas

for Your Business portfolio programs, including the SBES program. These measures included

LED lamps and fixtures, HP/RW T8s, LED exit signs, new T8/T5 fixtures, occupancy sensors,

and photocells. Navigant found the tracked ex ante savings from multi-family common area

measures summed to 5,582 MWh. Of this, ComEd allocated 1,639 MWh to the SBES program

and the remaining 3,943 MWh to the BILD program as overlap savings.

Recommendation 5. Navigant observed that ComEd may be tracking savings from multi-family

common areas separately for the SBES and BILD programs. The SBES program evaluation

team verified only the measures installed through the SBES program in PY7. Navigant

suggests that, if it becomes necessary for ComEd to track specific program metrics for savings



and BILD for these common measures.


coming from multi-family business sector, ComEd should track the savings and measures

installed in multi-family common areas under one program for evaluation.

Process Findings

Finding 6. Some participants and trade allies expressed frustration at the spending cap, which

caused the SBES program manager to restrict program starting in early January 2015.40 Some

trade allies maintained that stopping funding midyear had hurt their businesses, and the

issue was also mentioned by some participants. Trade allies and customers make plans

assuming that advertised SBES program measures will be available.

Recommendation 6. ComEd should explore ways to bring the demand for subsidized EE

measures offered to its small C&I customers through the SBES program into better balance

with the available budget. We understand that ComEd has already taken an initial step

toward this goal in PY8, by implementing a pre-application step that must be completed and

submitted by the trade ally prior to undertaking any SBES-approved work through the

program. This should provide some much-needed insight into the program’s burn rate –

previously the program did not know how much of the budget had been committed until

trade allies submitted their invoices. However, if the program continues to exceed its budget

level in the future, ComEd should consider reducing the incentive rates for SBES-approved

measures, or reducing the number of SBES-approved measures.

40 Letter from ComEd SBES Program manager to SBES trade allies dated January 5, 2015 (“Re: ComEd Smart Ideas for

Your Business ® Small Business Energy Savings (SBES) Program Budget and Remainder of PY7”).


6 Appendix

6.1 Evaluation Research Impact Approaches and Findings

6.1.1 Evaluation Research Gross Impact Parameter Estimates

As described in Section 2, gross energy and demand savings for lighting measures are estimated using

the following formula as specified in the TRM:

Verified Gross Annual kWh Savings = Program bulbs * Delta Watts/1000 * HOU * IEe* ISR

Verified Gross Annual kW Savings = Program bulbs * Delta Watts/1000 * ISR

Verified Gross Annual Peak kW Savings = Gross Annual kW Savings * Peak Load CF * IEd * ISR

Where:

Delta Watts = Difference between the Baseline Wattage and CFL Wattage

HOU = Annual Hours of Use

ISR = Installation Rate

Peak Load CF = Peak Load Coincidence factor is calculated as the percentage of program

bulbs turned on during peak hours (Central Time Zone weekdays from 1 to 5 p.m. for

summer and between 6am-8am and 5pm-7pm for winter).

IEe = Energy Interactive Effects

IEd = Demand Interactive Effects

Table 6-1below presents the measure level gross impact findings. Table 6-2 below presents the gross

impact findings by building type.

Table 6-1. Measure Level Gross Impact Results

Measure

Ex Ante Gross Savings (MWh)

Verified Gross Energy Savings (MWh)

Gross Energy RR

Notes on RR Recommendation Priority

Bath & Kitchen Aerators 10 19 194% Updated ex ante values to use TRM (V3.0)

Update tracking default values to use current approved version of TRM

Low

Vending Miser 58 58 100% No action NA Low

Showerheads 9 9 100% No action NA

Pre Rinse Sprayers 10 10 100% No action NA Low

Cooler Miser 21 21 100% No action NA Low

Beverage Machine Controls 79 79 100% No action NA Low

Reach-in (Novelty) Cooler Controls 181 182 100% No action NA

Snack Machine Controls 0.343 0.343 100% No action NA

Low T12 to HPT8/LW Retrofit 31,118 31,073 100%

2 projects defined with exterior space. Adjusted savings accordingly

NA

LED Exit Sign 1,161 1,163 100% No action?

Exterior LED 334 334 100% No action NA

Delamping T12 w/wo Reflector to HPT8

21,832 21,832 100% No action NA


Measure

Ex Ante Gross Savings (MWh)

Verified Gross Energy Savings (MWh)

Gross Energy RR

Notes on RR Recommendation Priority

4, 6-Lamp HPT8 w/HBF High-bay 6,470 6,470 100% No action NA

Occupancy Sensors 1,058 1,058 100% No action NA

GREM - PTAC 19 19 100% No action NA

Night Covers/Curtains 1 1 100% No action NA

Standard T8 to HPT8/RW 44,441 44,441 100% No action NA

2,3-Foot T8 Lamp and Ballast 838 841 100% Adjustment due to building type and TRM interactive factors

Ensure TRM inputs matches defined space in tracking system

Low

Removed 2,4,8-Foot Lamp w/wo Reflector

26,678 26,738 100% No action NA

LED Fixtures 19,756 19,964 101%

Changed space type savings input assumptions to match TRM e especially for religious space


Medium

New T8/T5 Fixtures with Electronic Ballasts

24,251 24,196 100%

Changed space type savings input assumptions to match TRM e especially for religious space


Medium

LED Refrigerated Display Case Lighting Open/Closed 305 304 100% Adjusted for grocery space

Use default values for defined space

Medium

Photocells 13 13 100% No action NA

Outdoor LED Channel Sign ≤ 2 Feet 6 6 100% No action NA

Induction Fixtures 28 28 100% No action NA

Daylighting Controls 2 2 100% No action NA

EC Motor for Walk-in Cooler or Freezer

456 456 100% No action NA

Anti-Sweat Heater Controls for Glass Door Cooler or Refrigerator

2,738 2,737 100% Minor adjust due to 2 projects with higher per unit savings

NA Low

Strip Curtains for Cooler/Freezer 536 242 45% ex ante over estimated savings using wrong custom calculation

Use TRM for savings per cooler and freezer with strip curtains instead of custom savings in square foot

High

Auto Closer for Walk-in Cooler/Freezer


Evaporator Fan Controls on EC Motor


Time Clocks for Lighting 2 2 100% No action NA

Source: Navigant Team Analysis


Table 6-2. PY7 Verified Gross Impact Savings Estimates by Building Type

Measure Ex Ante Gross

Savings (MWh)

Ex Ante Peak Demand

Savings (MW)

Verified Gross Energy Savings

(MWh)

Gross Energy Realization

Rate

Verified Peak Demand Savings

(MW)

Office 49,226 7.44 49,285 100% 7.44

Retail/Service 45,366 8.74 45,618 101% 8.74

Light Industry 15,731 2.71 15,732 100% 2.71

Miscellaneous 15,573 2.42 15,593 100% 2.42

Religious Organization 3,743 1.49 3,661 98% 1.49

Warehouse 29,209 4.13 29,217 100% 4.13

Restaurant 4,471 1.07 4,547 102% 1.07

Exterior 10 - 9 93% -

Grocery 9,053 1.11 11,219 124% 1.11

Low-User Small Business

8,858 1.92 8,838 100% 1.92

Hotel / Motel 51 0.00 52 102% 0.00

Multi-Family Common Area

1,639 0.20 1,639 100% 0.20

Garage, 24/7 Lighting 25 0.00 33 132% 0.00

Garage 1 0.00 1 98% 0.00

Total 182,959 31.22 185,445 101% 31.22



6.2 Detailed Process Findings

PY7 process evaluation activities involved information gathering on the effectiveness of the current

program design, administration, delivery, implementation processes, customer and program partner

experience and satisfaction, and opportunities for program improvement. The process analysis included a

synthesis of both qualitative and quantitative data collected during the program implementer and

program coordinator interviews, the end-user customer surveys and the trade ally in-depth interviews.

The major process topics include:

Effectiveness of program implementation and program changes

Administration and Delivery

Effectiveness of program design and processes

Customer and program partner experience and satisfaction with the program

Opportunities for program improvement

Navigant conducted rolling customer process surveys beginning with the latter part of PY6 and

continued through the first half of PY7, and process research interviews in the latter part of PY7 with the

trade allies.

6.2.1 Effectiveness of Program Implementation and Program Changes

6.2.1.1 Marketing

First Became Aware of the SBES Program

Customers participating in the SBES Program were equally likely to first hear about the program from

trade allies (46 percent) and professional or family contacts (46 percent).

Figure 6-1. . First Became Aware of the SBES Program

Source: CATI customer survey responses. Note: percentages do not sum to 100 percent because multiple

response were permitted.

46%

46%

9%

5%

3%

3%

2%

2%

1%

% 10% 20% 30% 40% 50%

Trade Ally

Word of Mouth

Program Energy Advisor

A Mailer

Landlord/manager

Other

ComEd Account Manager

ComEd Website

Email


Four trade allies reported that they became aware of the SBES Program through their relationship with

ComEd. Implementation contractors informed three trade allies about the program. Supply house and

business associates were the source of program awareness for two trade allies each. One trade ally

initiated contact because of his involvement with other ComEd programs and the last trade ally found

out about the program at a trade show.

Received Information or Heard About the SBES Program

Almost all of the program participants surveyed discussed the program with their trade ally (85 percent).

A majority of the customers said they heard about the program from a colleague or friend (54 percent).

The information from ComEd included: information in email (11 percent), ComEd newsletter (10

percent), a web site visit (9 percent), an account manager contact (4 percent), a ComEd event (2 percent)

and a ComEd Webinar (1 percent). Twenty-one percent said they were contacted by an energy advisor

from the implementation contractor.

Figure 6-2. Received Information or Heard About SBES Program



According to trade allies, customers heard about the Small Business Energy Efficiency Program in two

main ways: through the trade ally knocking on their door and from referrals. One program partner

reported that a handful of customers heard about the program from radio ads. All of the trade allies

received at least a few referrals from other customers.

Most of the customers who received marketing materials found the materials useful. Forty-six percent of

the program participants did not see any marketing materials. A small percentage (5 percent) saw the

materials and did not find them useful.

85%

54%

37%

21%

0%

20%

40%

60%

80%

100%

Discussed/Trade Ally Word of Mouth Contact w/ComEd Contacted by EnergyAdvisor


Figure 6-3. Customers Found Marketing Materials Useful

Source: CATI customer survey responses.

The customer and trade ally data shows the growing importance of word-of-mouth in both customer and

trade ally program awareness and the importance of the trade ally in marketing the program. Over three-

fourths of the trade allies explained how the program worked, according to customer reports.

Trade allies were involved in the SBES Program for an average of 3.25 years.

6.2.1.2 Increasing Program Awareness

Seven of the twelve trade allies reported that they don’t know how the IC can boost program awareness.

One program partner said that there is no need to boost program awareness because the program can’t

keep up with the customer need for it. Another wished there were fewer contractors, not more. The two

trade allies with ideas on how to boost program awareness said direct mailings and having shoes on the

ground talking to contractors.

All of the trade allies in our study participated in other ComEd program such as the Standard or the

Custom Program. All but one program partner refers customers to other ComEd business programs such

as BILD and RCX.

The implementation contractor provides appropriate marketing materials, such as flyers and brochures,

to the trade allies to help them successfully market the program to customers. One program partner

thought that the IC was not strong in this area.

49%

5%

46%

Useful Not useful Did not see any


Trade allies’ answers ranged from:

No changes are needed (3)

They do a good job now (2)

No answer/Don’t know (2)

Send free marketing materials

Have better communication and outreach like the standard program

Provide a database of customers who have not participated

Advertise the program more or balance funding.

Customers reported that emails and flyers/ads/mailings are the best ways to reach small business

customers like themselves. They mentioned other preferred marketing channels such as trade

allies/contractors or bill inserts. Three of the top four channels are marketing channels that are usually

implemented by ComEd.

Figure 6-4. Best Channels to Reach Small Business Customers



6.2.2 Administration and Delivery

6.2.2.1 Program Delivery

Ten of the trade allies actively market the program by knocking on the doors of their customers and

others. Two of the trade allies relied more heavily on referrals. Forty percent of the trade allies marketed

to both current and new customers. Thirty-three percent marketed mostly to new customers and twenty

five percent marketed to current customers.

Program partners used different methods for marketing the SBES Program. For instance, two trade allies

used a random process of contacting small businesses that might qualify for the program. Other filters

3%

5%

7%

11%

14%

18%

21%

21%

38%

45%

% 10% 20% 30% 40% 50%

Nexant energy advisor

Word of Mouth

Other

Personal Visit

ComEd Account Manager

Telephone

Bill inserts

Trade allies/contractors

Flyers/ads/mailings

E-mail


including square footage, size, electrical use, geography, existing customer status and through referrals.

One program partner has an internal telephone sales group that call and pre-qualifies customers. Another

program partner contacts any small business with an open sign on the door.

6.2.2.2 Days from Approval to Installation

Trade allies reported a wide range in the number of days from approval from the IC to equipment

installation (from three days to two months). Trade allies may need to order energy efficient equipment

depending on the scope of the project.

Figure 6-5. Days From Project Approval to Equipment Installation – Lower and Upper Bounds

Source: Navigant evaluation of trade ally interview data

6.2.2.3 Number of Days to Payment Receipt

Trade allies said that it took from a few weeks to six months to receive payment for the SBES project after

the paperwork was submitted and accepted.

0

10

20

30

40

50

60

1 2 3 4 5 6 7 8 9 10 11 12

1 0

7 7 7

0 0 0

28

14

30

237

1014 14 14 14

28

42

5660 60

Shortest Time Longest Time


Figure 6-6. Number of Days to Payment Receipt After Application Approved

Source: Navigant evaluation of customer survey data.

6.2.3 Effectiveness of Program Design and Processes

6.2.3.1 Trade Allies Program Barriers

Many trade allies found the SBES Program slightly problematic. Nine of the twelve trade allies mentioned

at least one barrier to program participation. These concepts are captured in the following comments:

The program could have been more responsive to customers’ needs.

Some contractors seemed to be in the program to make money and not improve the efficiency of

customers. (2)

The payment and application process is burdensome.

Right now it is very hard to get money from customers. Some customers expect the installations

to be free…

People feel that ComEd should pay for everything, which is perpetuated by some trade allies.

The reservation process takes too much time.

Incentive amounts have changed frequently, often in the same program year.

Sometimes people are not forthright about giving us the information we need for the program.

6.2.3.2 Customer Program Barriers

Half of trade allies were able to identify barriers they believed were experienced by customers in PY7:

Cost

Sometimes I have to convince the customers that the program is real.

Running out of funds. Some of our customers received late installations because the program ran

out of money. Some of the install times can also be a tight squeeze if parts need to be ordered.

The fact that the program ended abruptly. We had no indication that it would end. We had to tell

our customers that we wouldn't be able to do the job. Now they are better at communicating

about funding levels.

A misalignment of expectations regarding the permanency of the offer during the program year

0

50

100

150

200

1 2 3 4 5 6 7 8 9 10 11 12

140

1428

90 90 90

120 120

21 30 3042 45

60

180 180 180 180 180

Shortest Time Longest Time


Whether they can participate. Sometimes by the time they want to participate, the funds are not

there.

Five of the six comments relate to the same issue: that the program runs out of money mid-year. Trade

allies appreciate communications on the remaining program funding levels to better serve their

customers. Another issue for trade allies was that equipment for some projects needs ordering – its

availability is out of the control of the program partner.


6.3 PJM Data and Findings

Small Business Energy Savings Program (SBES)

Program Year 7 (PY7) – June 1, 2014 – May 31, 2015

Ex-Post Gross Peak Demand (MW) Savings

The PJM summer ex-post gross coincident peak demand savings was 30.91 MW.

The PJM winter ex-post gross coincident peak demand savings was 20.00 MW.

List parameters included in the ex-post gross peak demand calculation.

(a) PY7 program bulbs and hot water measures installed

(b) Non-coincident kW reduction

(c) kW of baseline equipment and replacement equipment

(d) Summer PJM coincidence factor (CF) defined by weekday’s 1-5pm Central Time Zone, between

June 1 and August 31, and non-holidays

(e) Winter PJM coincidence factor (CF) defined by weekdays between 6am-8am and 5pm-7pm

Central Time Zone, between January 1 and February 28, and non-holidays

(f) Demand interactive effect

(g) kW of baseline equipment during Performance Hours

(h) kW of replacement equipment during Performance Hours

(i) Installation Rate

For lighting measures, the algorithms used to calculate demand savings were:

(a) Non-coincident kW reduction = kW of baseline equipment - kW of replacement equipment

(b) PJM Coincident kW reduction = non-coincident kW savings * Coincidence Factor * Demand

Interactive Effect * Installation Rate

For non-lighting measures, the algorithms used to calculate demand savings were:

(c) PJM Coincident kW reduction = kW of baseline equipment during Performance Hours - kW of replacement

equipment during Performance Hours

ComEd’s coincident peak demand savings for both baseline and post retrofit conditions are defined as

the average demand kW savings for the 1 PM CPT to 5 PM CPT non-holiday weekday time period for

summer, and 6 AM CPT to 8 AM CPT and 5 PM CPT to 7 PM CPT non-holiday weekday time period for

winter.41 If this energy savings measure is determined to have weather dependency then the summer

peak kW savings are based on the zonal weighted temperature humidity index (WTHI) standard, and the

winter peak kW savings are based on the zonal wind speed-adjusted temperature (WWP) standards

posted by PJM (there is also PJM Zonal Winter Weather Standards similar to summer WTHI).42 The zonal

WTHI and WWP are the mean of the zonal WTHI values or WWP values on the days in which PJM peak

load occurred in the past sixteen years (1998-2014). This mean ComEd WTHI value is 81.6 demand

savings for summer is the difference in kW between the baseline and post retrofit conditions. Similarly,

41 The Winter Weather Standard is the dry bulb temperature adjusted (by 0.5 °F) for wind speed above 10 mph. The

measurements were for Hour Ending 19:00 on RTO peak days.” 42 This is in accordance with the PJM manual 18, PJM Capacity Market, effective October 16, 2015.


the ComEd WWP value is 14.5 demand savings for winter is the difference in kW between the baseline

and post retrofit conditions.

The IL TRM does not list winter peak coincidence factors and winter peak savings. ComEd did not track

winter peak savings in PY7. Based on the IL TRM summer coincidence factors, Navigant estimated 30.91

MW summer peak coincidence demand savings for PY7. Navigant used secondary research winter

coincidence factors to estimate 20.00 MW winter peak coincidence demand savings for the PY7 Small

Business Program.43

43 Winter peak coincidence factor for commercial lighting were taken from Navigant/Itron study (ComEd

Commercial Lighting Winter Peak CF Recommendations_2015_02_19.pdf). Winter peak coincidence factors for non-

lighting commercial measures were sourced from the Connecticut TRM (Connecticut Program Savings Document,

8th ed. for 2013 Program Year).


6.4 Survey Instruments

6.4.1 Process Survey Instrument

NICOR/COMED or INTEGRYS/COMED SMALL BUSINESS ENERGY SAVINGS PROGRAM

PARTICIPANT SURVEY

PY7 Final (8/14/2015)

Table 1: Small Business Energy Efficiency Program Survey Topics

Topics Research Questions

Process Module Satisfaction

Marketing and Outreach

Benefits and Barriers

Feedback and Recommendations

Firmographics Model Ownership

Type

Age

Number of employees

Participation Type = DirectInstall1, DirectInstall2, DirectInstall3, CI=Contractor Installed DI=Direct Install

Lighting or non = 1=yes 2=no

Measure List List of measures installed during the assessment

INTRODUCTION

[READ IF CONTACT=1]

Hello, this is _____ from Blackstone calling on behalf of ComEd. This is not a sales call. May I please

speak with <CONTACTNAME>?

Our records show that < COMPANY > installed energy efficient <MEASURE1, MEASURE2,

MEASURE3> through the Small Business Energy Savings Program sponsored by ComEd. We are calling

to do a follow-up study about < COMPANY >'s participation in this incentive program. I was told you're

the person most knowledgeable about this project. Is this correct? [IF NOT, ASK TO BE

TRANSFERRED TO MOST KNOWLEDGEABLE PERSON OR RECORD NAME & NUMBER.]

This survey will take about 15 minutes. Is now a good time? [If no, schedule call-back]

[READ IF CONTACT=0]

Hello, this is _____ from Blackstone calling on behalf of ComEd. I would like to speak with the person

most knowledgeable about the recent assessment and changes in lighting, cooling or other energy -

related equipment for your firm at this location.

[IF NEEDED] Our records show that < COMPANY > installed energy efficient < MEASURE1,

MEASURE2, MEASURE3 > and your contractor received an incentive from ComEd. We are calling to do

a follow-up study about your firm's participation in this incentive program, which is called the Small

Business Energy Savings Program. I was told you're the person most knowledgeable about this project. Is

that correct? [IF NOT, ASK TO BE TRANSFERRED TO MOST KNOWLEDGEABLE PERSON OR

RECORD NAME & NUMBER.]



SCREENING QUESTIONS

A1. Just to confirm, did < COMPANY > recently participate in the Small Business Energy

Savings Program offered by ComEd at <ADDRESS>?

IF MORE EXPLANATION IS NEEDED: This is a program where your business may have

received a free energy assessment, an offer of free energy savings products, and a report.

IF <DIRECTINSTALL1, DIRECTINSTALL2, DIRECTINSTALL3>=[CI (CONTRACTOR

INSTALLED) OR DI (DIRECT INSTALL)]: Program incentives were paid directly to

your contractor who implemented one or more energy saving capital improvement

projects or equipment improvements.

1 Yes, participated as described

2 Yes, participated but at another location

3 NO, did NOT participate in program [if this is answered, go to A2]

97 OTHER, SPECIFY [if this is answered, go to A2]

98 DON'T KNOW [if this is answered, go to A2]

99 REFUSED [if this is answered, go to A2]

[SKIP A2 IF A1=1, 2]

A2. Is it possible that someone else dealt with the energy-efficient product installation?

1 YES, SOMEONE ELSE DEALT WITH IT

2 NO

97 OTHER, SPECIFY

98 DON'T KNOW

99 REFUSED

[IF A2=1, ask to be transferred to that person. If not available, thank and terminate. If available, go back to

A1]

[IF A1=2,3, 97,98,99: Thank and terminate. Record disposition as “Could not confirm participation”.]

Before we begin, I want to emphasize that this survey will only be about the energy saving products and

services received through the Small Business Energy Savings Program at <ADDRESS>.

[IF <DIRECTINSTALL1, DIRECTINSTALL2, DIRECTINSTALL3>=[DI (DIRECT INSTALL)< ASK

QA0-QA1]

Direct Install Measures

QA0. Were you present when < COMPANY > was visited by a trade ally from the Small

Business Energy Savings Program who conducted an assessment of your facility's

energy saving opportunities? 1. YES


2. NO 98. DON'T KNOW

99. REFUSED

QA1. I am going to read a list of energy saving products that our records indicate were

installed in your facility or building. Please confirm which of the following were

installed during the energy assessment. Also, let me know how many were installed?

ASK ABOUT MEASURE1 IF DIRECTINSTALL1 = DI, MEASURE2 IF

DIRECTINSTALL2=DI, MEASURE3 IF DIRECTINSTALL3=DI.

Direct_install QA1 QA1_Num

No Cost Products

Yes, data

from

database

Yes,

confirmed

No, not

installed

DK/NA If Yes, How

many were

installed?

[Range 1-300]

13 W CFLs

20 W CFLs

23 W CFLs

Bathroom Faucet Aerators (elec)

Kitchen Faucet Aerators (elec)

Showerheads (elec)

Pre-Rinse Sprayer

Hot Water Temperature Reset

Vending Miser

Cooling Miser

PROCESS MODULE

I'd now like to ask you a few general questions about your participation in the Small Business Energy

Efficiency program.

Program Application Process

S0 How did you first hear about the Small Business Energy Efficiency Program? (SELECT ALL

THAT APPLY. DO NOT READ. PROMPT IF NECESSARY)

1. ComEd Account Manager

2. ComEd Website

3. Program Energy Advisor

4. Contractor/Trade Ally

5. Email

6. Friend/colleague/word of mouth


97. OTHER, SPECIFY

98. DON'T KNOW

99. REFUSED

S1b Who explained the program requirements to you? (SELECT ALL THAT APPLY. DO NOT

READ. PROMPT IF NECESSARY)


2. ComEd/ Website

3. Program Energy Advisor


5. Email


97. OTHER, SPECIFY

98. DON'T KNOW

99. REFUSED

S1c How would you rate the application process? Please use a scale of 0 to 10 where 0 is “very

difficult” and 10 is “very easy”. [SCALE 0-10; 98=Don't know, 99=Refused]

[ASK S1d IF S1c<4] S1d Why did you rate it that way? [RECORD ALL THAT APPLY]

1. Difficult to understand

2. Long process

3. I did not complete the application process

97. OTHER, SPECIFY

98. DON'T KNOW

99. REFUSED

[SKIP TO S11 IF DIRECTINSTALL1=DI AND IF DIRECTINSTALL2=DI AND DIRECTINSTALL3 = DI] Contractor Relationship

ASK IF [DIRECTINSTALL1 = CI (CONTRACTOR INSTALLED) OR DIRECTINSTALL2=CI OR DIRECTINSTALL3 = CI] S1 Was more than one contractor involved in installing your energy efficient equipment?

1. Yes 2. No


98. DON'T KNOW

99. REFUSED

S2 Would you describe the contractor who did most of the work as a lighting contractor?

1. Lighting contractor

2. Not a lighting contractor

98. DON'T KNOW

99. REFUSED

[ASK IF S2 = 2] S2A What type of contractor was he?

1. HVAC 2. Plumber

97. Other (Describe_____________________ [skip S3 if S1=2,98 or 99] S3 Would you describe the second contractor as a lighting contractor?

1. Lighting contractor

2. Not a lighting contractor

98. DON'T KNOW

99. REFUSED

[ASK IF S3 = 2]

S3A What type of contractor was he?

1. HVAC 2. Plumber 3. Sheet metal contractor 4. Engineer

97. Other (Describe_____________________

[ASK IF S2=2 OR S3 = 2. ELSE SKIP TO S5] S4 How would you rate the non-lighting contractor's ability to meet your needs in terms of

implementing your project? Please use a scale from 0 to 10, where 0 is “not at all able to meet

needs” and 10 is “completely able to meet needs”? [SCALE 0-10; 98=Don't know, 99=Refused]

S4a On a scale of 0 to 10, where 0 is very dissatisfied and 10 is very satisfied, how would you rate

your overall satisfaction with your non-lighting contractor? [SCALE 0-10; 96=not applicable,

98=Don't know, 99=Refused]


S5a Would you recommend this contractor to other people or companies?

1. Yes [GO TO S5 IF S1 =1]

2. No

98. DON'T KNOW [GO TO S5 IF S1 =1]

99. REFUSED [GO TO S5 IF S1 =1]

[Ask S5b if S5a=2] S5b Why not? [RECORD ALL THAT APPLY]

1. Too small

2. Did not complete the work

3. Did not clean-up work area

4. Poor quality work

5. Did not complete in a timely manner

97. OTHER, SPECIFY 98. DON'T KNOW 99. REFUSED

ASK S5 IF S2 OR S3 = 1, OTHERWISE SKIP TO S8. S5 How would you rate the lighting contractor's ability to meet your needs in terms of

implementing your project? Please use a scale from 0 to 10, where 0 is “not at all able to meet

needs” and 10 is “completely able to meet needs”? [SCALE 0-10; 98=Don't know, 99=Refused]

S5aa On a scale of 0 to 10, where 0 is very dissatisfied and 10 is very satisfied, how would you rate

your overall satisfaction with your lighting contractor? [SCALE 0-10; 96=not applicable, 98=Don't

know, 99=Refused]

S6a Would you recommend this contractor to other people or companies?

1. YES

2. NO

98. DON'T KNOW

99. REFUSED

[Ask S6b if S6a=2] S6b Why not? [RECORD ALL THAT APPLY]

1. Too small

2. Did not complete the work

3. Did not clean-up work area

4. Poor quality work

5. Did not complete in a timely manner



Calls to the Smart Ideas For Business Call Center

S8 During the course of your participation in the program, did you place any calls to the Smart Ideas

for Business Call Center? 1. YES 2. NO

98. DON'T KNOW

99. REFUSED

[ASK S9 IF S8=1] S9 On a scale of 0 to 10, where 0 is “very dissatisfied” and 10 is “very satisfied;” how would you rate

your satisfaction with the Call Center's ability to answer your questions? [SCALE 0-10; 98=Don't

know, 99=Refused]

[ASK S10 IF S9<4] S10 Why did you rate it that way? [RECORD ALL THAT APPLY]

1. Provided inconsistent information

2. Didn't understand the question

3. Hard to reach the right person/person with the answer


Satisfaction With Program Attributes

[ASK OF ALL RESPONDENTS. RANDOMIZE S11a through S11c] S11 On a scale of 0 to 10, where 0 is very dissatisfied and 10 is very satisfied, how would you rate

your satisfaction with… [SCALE 0-10; 96=not applicable, 98=Don't know, 99=Refused]

a. The incentive amount b. The communication you had with the Smart Ideas program staff c. The equipment offered by the program (If needed: this is the equipment that is eligible

for an incentive under the program) d. The Small Business Energy Efficiency program overall e. ComEd overall

[ASK S12a IF S11a<4] S12a You indicated some dissatisfaction with the incentive amount, why did you rate it this way?

[RECORD ALL THAT APPLY] (DO NOT READ)

1. Better rebates in other states

2. Too small

3. Equipment didn't qualify



[ASK S12b IF S11b<4] S12b You indicated some dissatisfaction with the communication you had with the Smart Ideas staff,

why did you rate it this way? [RECORD ALL THAT APPLY] (DO NOT READ)

1. Provided inconsistent information

2. Didn't understand the question

3. Hard to reach the right person/person with the answer


[ASK S12b IF S11c<4] S12c You indicated some dissatisfaction with the measures offered by the Small Business Energy

Efficiency Program, why did you rate it this way? [OPEN END; 98=Don't know, 99=Refused]

[ASK S12d IF S11d<4] S12d You indicated some dissatisfaction with the Program overall, why did you rate it this way?

[RECORD ALL THAT APPLY] (DO NOT READ)

1. Not as easy as other states

2. No clear guidance


[ASK S12e IF S11e<4] S12e You indicated some dissatisfaction with ComEd overall, why did you rate it this way? [RECORD

ALL THAT APPLY] (DO NOT READ)

1. Rates are too high

2. It took too long to get rebate

3. Poor customer service

4. Poor power supply/service



Marketing and Outreach

MK0 I'm now going to ask you about several specific ways in which you might have seen or heard

information about the Small Business Energy Savings program. Have you ever… [1=Yes, 2=No,

8=(Don't know), 9=(Refused)]

a. Received information about the program in your monthly utility bill? Yes No DK/Refused

b. Attended a ComEd customer event where the program was

discussed?

c. Discussed the program with a ComEd Account Manager?

d. Discussed the program with a Contactor or Trade Ally?

e. Seen information about the program on the ComEd Website?

f. Received information about the program in an Email?

g. Heard about the program from a colleague, friend or family

member?

h. Attended a meeting, seminar or workshop where the program was

presented?

i. Attended a webinar where the program was discussed?

j. Read about the program in a ComEd Newsletter?

k. Been directly contacted by a program implementer energy advisor?

MK1b How useful were the program's marketing materials in providing information about the

program? Would you say they were…

1. Very useful

2. Somewhat useful

3. Not very useful

4. Not at all useful

5. DID NOT SEE MARKETING MATERIALS

98. DON'T KNOW

99. REFUSED

[ASK MK1c IF MK1b=3, 4]

MK1c What would have made the materials more useful to you? [Record/answer UP TO 3]

1. More detailed information

2. Where to get additional information

97. OTHER, SPECIFY

98. DON'T KNOW

99. REFUSED

[ASK ALL PARTICPANTS]


MK2 In general, what is the best way of reaching companies like yours to provide information about

energy efficiency opportunities like the Small Business Energy Efficiency program?

[Record/answer UP TO 3]

1. Bill inserts

2. Flyers/ads/mailings

3. E-mail

4. Telephone


6. Nexant/Franklin Energy advisor

8. Trade allies/contractors

97. OTHER, SPECIFY

98. DON'T KNOW

99. REFUSED

Benefits and Barriers

B1a What do you see as the main benefits to participating in the Small Business Energy Efficiency

program? [Record/answer UP TO 3] (DO NOT READ)

1. Energy Savings/saving money

2. Good for the Environment

3. Lower Maintenance Costs

4. Better Quality/New Equipment

5. Rebate/Incentive

9. Able to make improvements sooner

97. OTHER, SPECIFY

98. DON'T KNOW

99. REFUSED

B1b What do you see as the drawbacks to participating in the program? [Record/answer UP TO 3] (DO

NOT READ)

1. Paperwork too burdensome

2. Incentives not high enough/not worth the effort

3. Program is too complicated

4. Cost of equipment

5. No drawbacks

97. OTHER, SPECIFY

98. DON'T KNOW

99. REFUSED


R2 How would you improve the Small Business Energy Efficiency Program? [Record/answer UP TO

4] (DO NOT READ)

1. Higher incentives


2. More measures

3. Greater publicity

4. Better Communication/Improve Program Information

8. Simplify application process

11. Quicker processing times

97. OTHER, SPECIFY

96. NO RECOMMENDATIONS

98. DON'T KNOW

99. REFUSED

Firmographics

I only have a few general questions left.

F1 BLANK

F2 Which of the following best describes the ownership of this location?

1. <COMPANY> owns and occupies this location 2. <COMPANY> owns this facility but it is rented to someone else 3. <COMPANY> rents this facility

98. DON'T KNOW

99. REFUSED

F6 And which of the following best describes thelocation? This location is…

1. <COMPANY>'s only location

2. One of several locations owned by <COMPANY>

3. The headquarters location of <COMPANY> with several locations

98. DON'T KNOW

99. REFUSED

F7a And which of the following best describes the ownership of the lighting system in this building?

1. My company owns the lighting system

2. The owner of the building owns the lighting system

97. OTHER _SPECIFY

98. DON'T KNOW

99. REFUSED

F7b And which of the following best describes the ownership of the HVAC system in this building?

1. My company owns the HVAC system

2. The owner of the building owns the HVAC system

97. OTHER _SPECIFY

98. DON'T KNOW

99. REFUSED


F4a How old is this facility? RECORD IN YEARS [NUMERIC OPEN END, 0 TO 150; 998=Don't

know, 999=Refused]

F5a How many employees, full plus part-time, are employed at this facility? [NUMERIC OPEN END,

0 TO 2000; 9998=Don't know, 9999=Refused]

That brings us to the end of my questions for you. On behalf of ComEd, we thank you for your time today.

If in reviewing my notes, I discover a point I need to clarify, is it all right if I follow-up with you by phone

or email? [IF YES, VERIFY PHONE NUMBER OR EMAIL Require only one field (either phone or email)]


6.4.2 Customer Net to Gross Survey Instrument

COMED SMALL BUSINESS ENERGY EFFICIENCY PROGRAM

PARTICIPANT SURVEY

PY6 FINAL (12/19/2014)

Table 1: Small Business Energy Savings Program Survey Topics

Topics Research Questions

Net-to-Gross Would the customer have installed the energy

efficient equipment without the program?

Spillover Module Did the SBES Program encourage the customer

to install energy efficient equipment without an incentive? Why?

Process Module Satisfaction

Firmographics Model

Ownership

Type

Age

Number of employees

Participation Type = Direct Install

Direct Install + Contractor Installed

Contractor Installed Only

Assessment Only

Enduse = Lighting

Electric Non-lighting:

Refrigeration

HVAC

Water Heating

Direct Install List of measures installed during the assessment


INTRODUCTION

[READ IF CONTACT=1]

Hello, this is _____ from Blackstone Group calling on behalf of ComEd. This is not a sales call. May I

please speak with <PROGRAM_CONTACT>?

Our records show that <COMPANY> installed energy efficient <ENDUSE> through the Small Business

Energy Savings Program sponsored by ComEd. We are calling to do a follow-up study about

<COMPANY>’s participation in this incentive program. I was told you’re the person most knowledgeable

about this project. Is this correct? [IF NOT, ASK TO BE TRANSFERRED TO MOST KNOWLEDGABLE

PERSON OR RECORD NAME & NUMBER.]


[READ IF CONTACT=0]

Hello, this is _____ from Blackstone Group calling on behalf of ComEd. I would like to speak with the

person most knowledgeable about the recent assessment and changes in lighting, cooling or other energy-

related equipment for your firm at this location.

[IF NEEDED] Our records show that <COMPANY> installed energy efficient <ENDUSE> and your

contractor received an incentive of <INCENTIVE AMOUNT> from ComEd. We are calling to do a follow-

up study about your firm’s participation in this incentive program, which is called the Small Business

Energy Savings Program. I was told you’re the person most knowledgeable about this project. Is that

correct? [IF NOT, ASK TO BE TRANSFERRED TO MOST KNOWLEDGABLE PERSON OR RECORD

NAME & NUMBER.]


SCREENING QUESTIONS

A1. Just to confirm, between June 1, 2013 and May 31, 2014 did <COMPANY> participate in the

Small Business Energy Savings Program offered by ComEd at <ADDRESS>?

IF MORE EXPLANATION IS NEEDED: This is a program where your business may have

received a free energy assessment, an offer of free energy savings products, and a report.

Program incentives were paid directly to your contractor who implemented one or more energy

saving capital improvement projects or equipment improvements and tune-ups. 1 Yes, participated as described

2 Yes, participated but at another location

3 NO, did NOT participate in program [if this is answered, go to A2]

00 Other, specify [if this is answered, go to A2]

98 Don’t know [if this is answered, go to A2]

99 Refused [if this is answered, go to A2]

[SKIP A2 IF A1=1, 2]

A2. Is it possible that someone else dealt with the energy-efficient product installation?

1 Yes, someone else dealt with it

2 No

00 Other, specify

98 Don’t know


99 Refused

[IF A2=1, ask to be transferred to that person. If not available, thank and terminate. If available, go back to

A1]

[IF A1=2, 3, 00, 98, 99: Thank and terminate. Record disposition as “Could not confirm participation”.]

Before we begin, I want to emphasize that this survey will only be about the energy saving products and

services received through the Small Business Energy Savings Program at <ADDRESS>.

Program Application Process

S0 How did you hear about the Small Business Energy Savings Program? (SELECT ALL THAT

APPLY. DO NOT READ. PROMPT IF NECESSARY)


5. ComEd Website


4. Email


97. OTHER, SPECIFY

98. DON'T KNOW

99. REFUSED

ASK S1 IF S0 = 3; ELSE GO TO N1

S1 Did your organization have a previous business relationship with the trade ally that told you

about the Small Business Energy Savings Program?

PY7 NET-TO-GROSS MODULE VARIABLES

Variables for the net-to-gross module:

<NTG> (B=Basic rigor level, S= Standard rigor level. All questions here are asked if the standard rigor

level is designated. Basic rigor level is designated through skip patterns)

<UTILITY> (ComEd/)

<ENDUSE> (Type of measure installed; from program tracking dataset)

<OTHERPTS> (Variable to be calculated based on responses. Equals 1- minus response to N3p.)

<FINCRIT1> (Variable to be calculated based on responses. Equals 1 if payback period WITHOUT

incentive is shorter than company requirement. See instructions below.)

<FINCRIT2> (Variable to be calculated based on responses. Equals 1 if payback period WITH incentive is

shorter than company requirement. See instructions below.)

<MSAME> (Equals 1 if same customer had more than one project (multiple facilities) of the same end-use

type; from program tracking database)

<NSAME> (Number of additional projects (at other facilities) of the same end-use type implemented by

the same customer; from program tracking database)

<FSAME> (Equals 1 if same customer also had a measure of a different end-use type at the same facility;

from program tracking database)


<FDESC> (Type of end-use of a different measure type at the same facility; from program tracking

database)

NET-TO-GROSS BATTERY

I’d now like to ask a few questions about the <ENDUSE> you installed through the program.

N1 When did you first learn about ComEd’s Small Business Energy Savings Program? Was it

BEFORE or AFTER you first began to THINK about installing this equipment? (NOTE TO

INTERVIEWER: “this equipment” refers to the specific energy efficient equipment installed

through the program.) [Identical to Standard.]

1 Before

2 After

8 Don't know

9 Refused

[ASK N2 IF N1=2, 8, 9]

N2 Did you learn about ComEd's Program BEFORE or AFTER you DECIDED to implement the

equipment that was installed? (NOTE TO INTERVIEWER: “the equipment” refers to the specific

energy efficient equipment installed through the program.) [Identical to Standard.]

1 Before

2 After

8 Don't know

9 Refused

N3 Now I’m going to ask you to rate the importance of the program factors that might have

influenced your decision to install this equipment. Think of the degree of importance as being

shown on a scale with equally spaced units from 0 to 10, where 0 means not at all important and

10 means extremely important. Now, using this scale please rate the importance of each of the

following in your decision to implement the equipment at this time. [FOR N3a-f, RECORD 0 to

10; 96=Not Applicable; 98=Don’t Know; 99=Refused]

(If needed: How important in your DECISION to implement the project was…)

N3a Availability of the Small Business Program incentive

N3b Information provided through the technical assistance you received from the trade ally during

the energy assessment

N3c Recommendation from an equipment vendor or contractor that helped you with the choice of the

equipment

N3d Recommendation from a ComEd program staff person

N3e Information from the Small Business Energy Savings Program’s or ComEd’s marketing materials

N3f Information in your assessment report

N4 Next, I’m going to ask you to rate the importance of any factors OUTSIDE THE PROGRAM that

might have influenced your decision to install this equipment. Using the same zero to 10 scale,

where 0 means not at all important and 10 means extremely important, please rate the


importance of each of the following in your decision to implement the equipment at this time.

[FOR N4a-d, RECORD 0 to 10; 96=Not Applicable; 98=Don’t Know; 99=Refused]

(If needed: How important in your DECISION to implement the project was…)

N4a Concern about environmental issues

N4b Recommendation from a friend

N4c Recommendation from a vendor or contractor not affiliated with the Small Business Energy

Savings program or ComEd

N4d Age or condition of the old equipment

N4e Previous experience with this type of equipment

N5 Were there any other factors we haven't discussed that were important in your decision to install

this energy efficient equipment?

00 [Record verbatim] [Analyst scores after the fact to determine whether this is a program or

non-program influence.]

96 Nothing else was important

98 Don’t Know

99 Refused

[ASK N5a IF N5=00]

N5a Using the same zero to ten scale, how would you rate the influence of this factor? [RECORD 0 to

10; 98=Don’t Know; 99=Refused]

N6 Thinking about this differently, I would like you to compare the importance of the ComEd Small

Business Energy Savings Program with the importance of other factors in your decision to install

the <ENDUSE>. If you were given a TOTAL of 100 points that reflect the importance in your

decision to implement the <ENDUSE>, and you had to divide those 100 points between the

program on one hand, and other factors on the other, how many points would you give to the

importance of the PROGRAM?

Points given to program: [RECORD 0 to 100; 998=Don’t Know; 999=Refused]

[CALCULATE VARIABLE “OTHERPTS” AS: 100 MINUS N6 RESPONSE; IF N6=998, 999, SET

OTHERPTS=BLANK]

N6a And how many points would you give to other factors? [RECORD 0 to 100; 998=Don’t Know;

999=Refused] [The response should be equal to <OTHERPTS> because the number should sum

with <N6 RESPONSE> to equal 100. If response is not <OTHERPTS> ask INC1]

INC1 The last question asked you to divide a TOTAL of 100 points between the program and other

factors. You just noted that you would give <N6 RESPONSE> points to the program. Does that

mean you would give the other <OTHERPTS> points to other factors?

1 Yes

2 No

98 Don’t know

99 Refused


[IF INC1=2, go back to N6]

CONSISTENCY CHECK ON PROGRAM IMPORTANCE SCORE

[ASK IF (N6>69 AND ALL OF (N3a, N3b, N3c, N3d, N3e, N3f)=0,1,2,3), ELSE SKIP TO N7aa]

You just gave <N6 RESPONSE> points to the importance of the program. I would interpret that to mean

that the program was quite important to your decision to install this equipment. Earlier, when I asked

about the importance of individual elements of the program, such as the incentive, information provided

in the technical assessment report, and recommendations from ComEd program staff, I recorded some

answers that would imply that they were not that important to you. Just to make sure I have recorded

this properly, I have a couple of questions to ask you.

N7a When asked about THE AVAILABILITY OF THE PROGRAM INCENTIVE, you gave a rating of

...<N3a RESPONSE> ... out of ten, indicating that the program incentive was not that important to

you. Can you tell me why the incentive was not that important?

00 [Record VERBATIM]

98 Don't know

99 Refused

N7b When I asked you about THE INFORMATION IN THE TECHNICAL ASSESSMENT REPORT,

for instance, you gave a rating of ...<N3f RESPONSE> ... out of ten, indicating that the information

provided was not that important to you. Can you tell me why the information provided was not

that important?


98 Don't know

99 Refused

N7c When I asked you about THE RECOMMENDATION FROM A TRADE ALLY, you gave a rating

of ...<N3c RESPONSE> ... out of ten, indicating that the information provided was not that

important to you. Can you tell me why the information provided was not that important?


98 Don't know

99 Refused

N7d When asked about THE INFORMATION from the <PROGRAM> or <UTILITY> MARKETING

MATERIALS, you gave a rating of ...<N3e RESPONSE> ... out of ten, indicating that this

information from the program or utility marketing materials was not that important to you. Can

you tell me why this information was not that important?


98 Don't know

99 Refused


[ASK IF N6<31 AND ANY ONE OF (N3a, N3b, N3c, N3d, N3e OR N3f=8, 9, 10) ELSE SKIP TO N5]

N7aa You just gave <N6 RESPONSE> points to the importance of the program. I would interpret that to

mean that the program was not very important to your decision to install this equipment. Earlier,

when I asked about the importance of individual elements of the program I recorded some

answers that would imply that they were very important to you. Just to make sure I understand,

would you explain why the program was not very important in your decision to install this

equipment?


98 Don't know

99 Refused

Now I would like you to think about the action you would have taken with regard to the installation of

this equipment if the ComEd Small Business Energy Savings program had not been available.

N8 Using a likelihood scale from 0 to 10, where 0 is “Not at all likely” and 10 is “Extremely likely”, if

the ComEd Small Business Energy Savings program had not been available, what is the

likelihood that you would have installed exactly the same equipment? [RECORD 0 to 10; 98=

Don't know; 99=Refused]

CONSISTENCY CHECKS

[ASK N8a-d IF N3a=8,9,10 AND N8=7, 8, 9, 10]

N8a When you answered ...<N3A RESPONSE> ... for the question about the influence of the incentive,

I would interpret that to mean that the incentive was quite important to your decision to install.

Then, when you answered <N8 RESPONSE> for how likely you would be to install the same

equipment without the incentive, it sounds like the incentive was not very important in your

installation decision.

I want to check to see if I am misunderstanding your answers or if the questions may have been

unclear. Will you explain the role the incentive played in your decision to install this efficient

equipment?


98 Don't know

99 Refused

N8b Would you like for me to change your score on the importance of the incentive that you gave a

rating of <N3B RESPONSE>, or would you like to change your rating on the likelihood you

would install the same equipment without the incentive, which you gave a rating of <N8

RESPONSE>? We can change both if you wish.

1 Change importance of incentive rating

2 Change likelihood to install the same equipment rating

3 Change both

4 No, don’t change

8 Don't know

9 Refused


[ASK IF N8b=1 OR 3]

N8c How important was availability of the Small Business Energy Savings Program incentive? (IF

NEEDED: in your DECISION to implement the project) [Scale of 0 to 10, where 0 means not at all

important and 10 means extremely important];

98=Don't know

99=Refused]

[ASK IF N8b=2 OR 3]

N8d If the ComEd Small Business Energy Savings program had not been available, what is the

likelihood that you would have installed exactly the same equipment? [Scale of 0 to 10, where 0

means “Not at all likely” and 10 means “Extremely likely”?

98=Don't know

99=Refused]

[ASK IF N8>0, ELSE SKIP TO N9]

N8e You indicated earlier that there was a <N8 RESPONSE> in 10 likelihood that you would have

installed the same equipment if the program had not been available. Without the program, when

do you think you would have installed this equipment? Would you say…

1 At the same time

2 Earlier

3 Later

4 Never

8 Don't know

9 Refused

[ASK N8f IF N8e=3]

N8f How much later would you have installed this equipment? Would you say…

1 Within 6 months?

2 6 months to 1 year later

3 1 - 2 years later

4 2 - 3 years later

5 3 - 4 years later

6 4 or more years later

8 Don't know

9 Refused

[ASK N8g IF N8f=6]

N8g Why do you think it would have been 4 or more years later?


98 (Don't know)

99 (Refused)


N8h Could you please tell me in your own words what influence the ComEd Small Business Energy

Savings Program had in your DECISION to install the energy efficient equipment?


98 (Don't know)

99 (Refused)

PY4 SPILLOVER MODULE

Thank you for discussing the new <ENDUSE> that you installed through the Small Business Energy

Savings program. Next, I would like to discuss any energy efficient equipment you might have installed

OUTSIDE of the program.

SP1 Since your participation in the ComEd program, did you implement any ADDITIONAL energy

efficiency measures at this facility or at your other facilities within ComEd’s service territory that

did NOT receive incentives through any utility or government program?

1 Yes

2 No

8 Don't know

9 Refused

[ASK SP2-SP7i IF SP1=1, ELSE SKIP TO S0]

SP2 What was the first measure that you implemented? (IF RESPONSE IS GENERAL, E.G.,

“LIGHTING EQUIPMENT”, PROBE FOR SPECIFIC MEASURE. PROBE FROM LIST, IF

NECESSARY.)

1 Lighting: T8 lamps


3 Lighting: High Bay Fixture Replacement

4 Lighting: CFLs

5 Lighting: Controls / Occupancy sensors

6 Lighting: LED lamps

7 Cooling: Unitary/Split Air Conditioning System

8 HVAC: Packaged Terminal air conditioners or heat pumps

9 Cooling: Room air conditioners

10 Heating: Furnace

11 Heating: Boiler

12 Variable Frequency Drives (VFD/VSD) on HVAC Motors

13 Programmable Thermostat

14 Refrigeration LED Case Lighting

15 Refrigeration EC motor for cooler/freezer

16 Wall or roof insulation

17 New windows

18 Water heater

00 Other, specify

96 Didn’t implement any measures


98 Don't know

99 Refused

[SKIP TO S0 (PROCESS MODULE) IF SP2=96, 98, 99]

SP3 What was the second measure? (IF RESPONSE IS GENERAL, E.G., “LIGHTING EQUIPMENT”,

PROBE FOR SPECIFIC MEASURE. PROBE FROM LIST, IF NECESSARY.)



3 Lighting: High Bay Fixture Replacement

4 Lighting: CFLs

5 Lighting: Controls / Occupancy sensors

6 Lighting: LED lamps

7 Cooling: Unitary/Split Air Conditioning System

8 HVAC: Packaged Terminal air conditioners or heat pumps

9 Cooling: Room air conditioners

10 Heating: Furnace

11 Heating: Boiler

12 Variable Frequency Drives (VFD/VSD) on HVAC Motors

13 Programmable Thermostat

14 Refrigeration LED Case Lighting

15 Refrigeration EC motor for cooler/freezer

16 Wall or roof insulation

17 New windows

18 Water heater

00 Other, specify

96 Didn’t implement any measures

98 Don't know

99 Refused

SP4 BLANK

SP5 I have a few questions about the FIRST type of equipment that you installed. (If needed, read

back measure: <SP2 RESPONSE>) [OPEN END]

a. Why did you not receive an incentive for this equipment?

b. Why did you not install this equipment through the Small Business Program?

c. Please describe the SIZE, TYPE, and OTHER ATTRIBUTES of this equipment.

d. Please describe the EFFICIENCY of this equipment.

e. How many of these did you install?

SP5g. How significant was your experience in the ComEd Program in your decision to implement this

equipment, using a scale of 0 to 10, where 0 is not at all significant and 10 is extremely

significant? [SCALE 0-10; 98=Don’t Know; 99=Refused]


[SKIP SP5h IF SP5g = 98, 99]

SP5h. Why do you give it this rating? [OPEN END]

SP5i. If you had not participated in the ComEd Small Business Energy Savings program, how likely is

it that your organization would still have installed this equipment, using a 0 to 10, scale where 0

means you definitely WOULD NOT have installed this equipment and 10 means you definitely

WOULD have installed this equipment? [SCALE 0-10; 98=Don’t Know; 99=Refused]

[SKIP SP6-SP7i IF SP3=96, 98, 99]

SP6 I have a few questions about the SECOND type of equipment that you installed. (If needed, read

back equipment type: <SP3 RESPONSE>) [OPEN END]

a. Why did you not receive an incentive for this equipment?

b. Why did you not install this equipment through the <UTILITY> Program?

c. Please describe the SIZE, TYPE, and OTHER ATTRIBUTES of this equipment.

d. Please describe the EFFICIENCY of this equipment.

e. How many of these did you install?

SP6g. How significant was your experience in the ComEd Program in your decision to install this

equipment, using a scale of 0 to 10, where 0 is not at all significant and 10 is extremely

significant? [SCALE 0-10; 98=Don’t Know; 99=Refused]

[SKIP SP6h IF SP6g = 98, 99]

SP6h. Why do you give it this rating? [OPEN END]

SP6i. If you had not participated in the ComEd Small Business Energy Savings program, how likely is

it that your organization would still have installed this equipment, using a 0 to 10, scale where 0

means you definitely WOULD NOT have installed this equipment and 10 means you definitely

WOULD have installed this equipment? [SCALE 0-10; 98=Don’t Know; 99=Refused]

[ASK OF ALL RESPONDENTS] S11 On a scale of 0 to 10, where 0 is very dissatisfied and 10 is very satisfied, how would you rate

your satisfaction with… [SCALE 0-10; 96=not applicable, 98=Don’t know, 99=Refused]

a. The incentive amount b. The communication you had with the Smart Ideas program staff c. The measures offered by the program (If needed: this is the equipment that is eligible for

an incentive under the program) d. The Small Business Energy Savings program overall e. ComEd overall

S12 Has your organization participated in other ComEd programs in the past?

1 Yes

2 No

8 Don't know

9 Refused

ASK IF S12=1.


S13 Did you participate in the

1. Small Business Program

2. Standard Program

3. Custom Program

4. Other (Specify__________

8 Don’t know

9 Refused


R2 How would you improve the Small Business Energy Savings Program? [Record/answer UP TO 4]

1. Higher incentives

2. More measures

3. Greater publicity

4. Better Communication/Improve Program Information

8. Simplify application process

11. Quicker processing times

00. Other, specify

96. No recommendations

98. Don’t know

99. Refused

Firmographics

I only have a few general questions left.

F1 BLANK

F2 Which of the following best describes the ownership of this facility?

1. <COMPANY> owns and occupies this facility 2. <COMPANY> owns this facility but it is rented to someone else 3. <COMPANY> rents this facility

8. Don’t know

9. Refused

F6 And which of the following best describes the facility? This facility is…

1. <COMPANY>’s only location

2. One of several locations owned by <COMPANY>

3. The headquarters location of <COMPANY> with several locations

8. Don’t know

9. Refused

F7a And which of the following best describes the ownership of the lighting system in this building?

1. My company owns the lighting system


2. The owner of the building owns the lighting system

3. Other _Specify

8. Don’t know

9. Refused

F7b And which of the following best describes the ownership of the HVAC system in this building?

1. My company owns the HVAC system

2. The owner of the building owns the HVAC system

3. Other _Specify

8. Don’t know

9. Refused

F4a How old is this facility? [NUMERIC OPEN END, 0 TO 150; 998=Don’t know, 999=Refused]

F5a How many employees, full plus part-time, are employed at this facility? [NUMERIC OPEN END,

0 TO 2000; 9998=Don’t know, 9999=Refused]

That brings us to the end of my questions for you. On behalf of ComEd, we thank you for your time today.

If in reviewing my notes, I discover a point I need to clarify, is it all right if I follow-up with you by phone

or email? [IF YES, VERIFY PHONE NUMBER OR EMAIL]


6.4.3 Contractor In-Depth Interview Guide

ComEd Evaluation for the Small Business Program Contractor In-Depth Interview Guide September 17, 2015

Section Topics Questions

Background

What type of business does the trade ally conduct and

what types of experience does this trade representative

have? Q1-Q4

Free Ridership and Spillover

Would small business customers have installed the

equipment without the program (free ridership)? About

what percentage of customers have installed additional

energy efficient equipment without an incentive

(spillover)? Have they encouraged customers to

implement measures or behavior changes for which there

is no incentive? If so, do they know if the customers have

done so?

FR1-S5

Marketing and Participation

How did trade ally become aware of this program? Do

you refer customers to other utility programs? Is the level

of utility marketing sufficient? How can the Implementer

support your program marketing? Has word of mouth

marketing had an impact?

Q5-Q14

Program Barriers

What are the barriers to participation encountered by

customers and trade allies? How could the program be

changed to overcome these? Q15-Q19

Delivery and Administration (Contractor Perspective)

How do you market the program? Does program

delivery occur in a timely manner? Are you able to

provide qualified customers with a loan arrangement? Q20-Q26

Program Satisfaction

How satisfied are trade allies with the program? What do you like best about the program? Least? IF you could change one thing, what would it be? How satisfied are customers with the program?

Q27-Q32

Economic Indicators

Have your business revenues grown? Have you hired

more employees? Q33-Q36


[Note to Reviewer] The Interview Guide is a tool to guide process evaluation interviews. The guide helps

to ensure the interviews include questions concerning the most important issues being investigated in this

study. Follow-up questions are a normal part of these types of interviews. Therefore, there will be sets of

questions that will be more fully explored with some individuals than with others. The depth of the

exploration with any particular respondent will be guided by the role that individual played in the program,

i.e., where they have significant experiences for meaningful responses. The interviews will be audio taped

and transcribed.

Introduction (Note: the interviewer should change the introduction to match his/her own interviewing style)

Hi, may I please speak with [NAME]?

My name is ___ and I’m calling from Navigant Consulting. We are part of the team hired to

conduct an evaluation of the ComEd Small Business Energy Efficiency Program. At this time we

are interested in asking you some questions about your experiences with the Small Business

Program. The questions will only take about a half hour. Is this a good time to talk? [IF NOT,

SCHEDULE A CALL BACK.]

I want to let you know that this call will be recorded for quality control purposes. Responses will

remain confidential and only be reported in aggregate with other responses.

We are evaluating last year’s program that began June 1, 2014 and ended May 31, 2015.

Background

1. Can you briefly describe the company you work for and the type of business it conducts?

2. How many full-time employees are employed at your company?

3. Do you mainly serve small businesses, large businesses or a mix of the two?

4. Can you briefly summarize your roles and responsibilities at your company?

FREE RIDERSHIP We are trying to understand the impact of the program separate from other factors that are at work in

your market. We have a number of questions that ask you to think about the Small Business program’s

effect on your ability to sell in this market.

Program Components Score

FR1. On a scale where 0 is not at all influential, 10 is very influential, how would you rate the program

rebates in helping you convince your customers to buy energy efficient measures??

___________________

FR2a. On a scale where 0 is not at all influential, 10 is very influential, how would you rate the Energy

Advisor in helping you convince your customers to buy energy efficient measures?

_______________________


FR2b. On a scale where 0 is not at all influential, 10 is very influential, how would you rate the

potential energy bill savings in helping you convince your customers to buy energy efficient

measures?

_______________________

FR3. ? On a scale where 0 is not at all influential, 10 is very influential, how would you rate all the

program features combined in helping you convince your customers to buy energy efficient

measures?

________________________

Program Components Score = Maximum of (FR1, FR2a, FR3).

Program Influence Score

According to our data, your company was associated with <x> customers [or <x> projects] that went

through the program from June 2014 through May of 2015. Your data also indicate that these customers

achieved <x> kWh of savings from their projects.

FR4. What percent of these kWh savings do you think those customers would have achieved if the

program had not been there?

[Program Influence Score = 100% - FR4%]

FREE RIDERSHIP ALGORITHM

Free Ridership = 1 - Average (Program Components Score/10, Program Influence Score)

[Consistency Check. Interviewer uses to resolve inconsistencies. Analyst uses to adjust scores if

inconsistencies remain.]

FR5. In your own words, please tell me what role the program played in getting these projects

installed. [Open ended]

[For open-ended interviews, circle back to resolve discrepancies.]

[Informational question, not in the algorithm]

Sometimes utility energy efficiency programs help markets for energy efficiency services grow. They may

help some service providers more than others.

FR6. What percent of your customers would have gone to one of your competitors to do the same

project if the program had not been there? SPILLOVER

S1. Approximately what percent of your small business customers purchased equipment

that would have qualified for an incentive but did not apply for the incentive offered by

the utility last program year? [PROVIDE EXAMPLE IF NEEDED]

1) [RECORD VERBATIM RESPONSE]

88) Don’t know

99) Refused

[ONLY ASK IF S1 > 0]

a. On a scale of 0 to 10 where 0 means NOT AT ALL INFLUENTIAL and 10 means

EXTREMELY INFLUENTIAL, how would you rate the influence of the Small

Business Energy Efficiency Program on the customer decision to install this

equipment without an incentive?


1) [RECORD NUMERIC RESPONSE]

88) Don’t know

99) Refused

SPILLOVER ALGORITHM

(1) Spillover TAx = [If Program Influence Score for TAx >7] Eligible Non-

Incented kWhsTAx

(2) Eligible Non-Incented kWhTAx = Total Eligible kWhTAx * (S1TAx)

(3) Total Eligible kWhTAx = Program Incented kWhTAx / (1-S1Tax)

(4) Eligible Non-Incented kWhTAx = [Program Incented kWhTAx / (1-S1Tax)] *

S1TAx

b. What types of equipment did the customer(s) install without incentives?

(SUGGESTION: PROBE FOR THIS IF YOU CAN, FOCUS IS ON GAS EQUIPMENT)


88) Don’t know

99) Refused

c. [ADDED Q] How did the program influence sales of equipment installed without

incentives?


88) Don’t know

99) Refused

d. What were the reasons that they did not apply for the incentive? (e.g., too time-

consuming, too much paperwork, incentive too small to bother)


88) Don’t know

99) Refused

[ASK S2 IF S1=88] [That is, if the participant cannot quantify Spillover in percentage terms, ask about

number of projects]

S2. In the last year, did any of your small business customers install equipment that was eligible for a

Small Business Energy Efficiency Program rebate but did not receive a that rebate?

1) Yes

2) No

88) Don’t know


99) Refused

[ASK S3 IF S2=1]

S3. Approximately, how many of your projects last year were eligible for a Small Business Energy

Efficiency Program rebate but did not receive a rebate? [NUMERIC OPEN END; 88=DON’T

KNOW]

[ONLY ASK IF S3 > 0]

a. On a scale of 0 to 10 where 0 means NOT AT ALL INFLUENTIAL and 10 means

VERY INFLUENTIAL, how would you rate the influence of the Small Business

Energy Efficiency Program on the customer decision to install this equipment

without an incentive?

1) [RECORD NUMERIC RESPONSE]

88) Don’t know

99) Refused

S4. [ADDED Q] Have you encouraged small businesses to install other energy efficiency

equipment without an incentive from the program as a result of your participation in the

program?

S5 [ADDED Q] Have you encouraged small business customers to implement energy

saving behavior changes for which there is no incentive? [Example: changing furnace

filters, water heater temperature setback] [If yes]: How many [or what percent] of your

small business customers follow your recommendation?

Intro:

Now, I would like to ask you about the marketing of the Small Business Program.

Marketing

5. How did you (the contractor) become aware of the program?

6. How many years has your company participated in this program?

7. What ways can the utilities and program implementers boost program awareness with

contractors?

8. Do you participate in any other ComEd rebate programs such as Standard or Custom

Programs?

9. Have you referred any Small Business customers to other ComEd business programs?

10. What kind of support, if any, does Nexant, the implementer, provide for marketing the

Small Business Program to your customers?

11. How can the implementer more effectively support your program marketing?

12. Do you think promotional efforts are successful? Do you think the level of marketing and

promotion of the Small Business Program has been appropriate so far?

13. How do your customers hear about the program?


14. Did notice any spontaneous word- of- mouth marketing among ComEd customers? For

example, do customers know of other participating businesses?

Program Characteristics and Barriers

15. What areas could be improved to create a more effective program for customers and

program partners such as yourself?

16. What would you like to see added to the program’s contractor approved equipment list?

17. What barriers have you encountered with the program?

18. What barriers have customers encountered?

19. What program changes could be made to reduce trade ally or customer barriers?

Delivery and Administration (Contractor Perspective)

20. Do you actively market the program to your customers?

21. How did you decide which ComEd customers to contact about the program?

22. Are these businesses existing customers of yours?

23. After the customer agrees to install the recommended equipment, how long does it usually

take to schedule the installation?

24. How long did it take the implementer to process your payment after the paperwork was

submitted and accepted?

25. Were you able to provide qualified customers with a loan arrangement? Who financed

these loans?

26. About what percent of your Small Business program sales are financed? What percent of

customers requested financing?

Satisfaction with the Small Business Program

27. Are you satisfied with the program? Why or why not?

28. What do you like best about the Program?

29. What do you least like about the Program?

30. Are customers satisfied with the program? Why or why not?

31. Are the incentives levels effective at encouraging customers to install equipment they would

not have considered without the program?

32. If you could change one thing, what would it be?


Economic Indicators

33. Do you think the Small Business Energy Efficiency Program is a competitive advantage for

your firm?

34. Have your business revenues grown in the past year because of the Small Business

Program? About what percent?

35. Have you hired more employees in the past year because of work generated by the Small

Business Program? How many?

36. In this year will you hire more employees to handle increased work generated by the

program? About how many?

CLOSING SECTION

That brings us to the end of my questions. Is there anything else that you would like to let us know based

on the topics we covered today?

On behalf of ComEd we thank you for your time today. If in reviewing my notes, I discover a point I need

to clarify, is it all right if I follow-up with you by phone or email? [IF YES, VERIFY PHONE NUMBER OR

EMAIL ADDRESS]

SMALL BUSINESS ENERGY SAVINGS PROGRAM ......2016/02/13 · ComEd Small Business Energy Savings PY7...

Documents

Transcript of SMALL BUSINESS ENERGY SAVINGS PROGRAM ......2016/02/13 · ComEd Small Business Energy Savings PY7...