Can High Bill Alerts Help Utility Customers to Save Energy? · 2019-08-02 · • Reduce customer...

18
Ryan Fulleman Can High Bill Alerts Help Utility Customers to Save Energy? IEPEC Denver, CO August 20, 2019

Transcript of Can High Bill Alerts Help Utility Customers to Save Energy? · 2019-08-02 · • Reduce customer...

  • Ryan Fulleman

    Can High Bill Alerts Help Utility Customers to Save Energy?

    IEPEC Denver, COAugust 20, 2019

  • IEPEC, 2019 Denver

    Agenda

    HBA Pilot Evaluation ApproachFindingsTakeaways and Future Research

    2

  • IEPEC, 2019 Denver

    HBA Pilot Overview

    • Xcel Energy sought to reduce call center call volumes• Many residential customers call about higher-than-expected bills• Reduce customer service costs

    • High Bill Alert Pilot Program• 50,000 MN residential dual-fuel customers automatically enrolled• Email alert sent when customer’s monthly electricity or gas consumption

    was on track to exceed normal levels• Alerts sent June 2015 - June 2016 • HER customers excluded• Implemented by Opower (Oracle)

    3

  • IEPEC, 2019 Denver

    High Bill Alert Email

    4

    Bill alert when spending on track to be 30% higher than normal

    Recipient directed to Xcel Energy’s on-line Home Energy Assessment website

    Customers can opt out

    Features

    Xcel Energy branding

  • IEPEC, 2019 Denver

    High Bill Alert Delivery

    5

    Electric HBAs Gas HBAs

    • 59% of HBA emails were opened by recipient• 6.4% of HBA emails resulted in clicking the Xcel Energy on-line energy audit link

    FuelNumber of

    Customers Sent HBAs

    Number of HBAs Sent

    Average Number of HBAs per Treated

    CustomerElectricity 31,591 100,320 3.2Natural Gas 6,163 13,355 2.2

    Sheet1

    Electricity Savings

    Phase 1 (June 2015 – May 2016)Phase 2 (June 2016 – April 2018)

    Model 1Model 2Model 1Model 2Phase 1Phase 2

    Average Daily Savings per Customer (kWh)0.0770.0770.1330.134% Savings0.4%0.6%

    (0.043)(0.043)(0.057)(0.057)kWh Savings0.0770.134

    Customer Fixed EffectsYesYesYesYesBaseline21.388888888923.1034482759

    Weather VariablesNoYesNoYes

    Month-Year Fixed EffectsYesYesYesYes% Savings Error Margin0.33%0.41%

    Number of Customers75,26775,26775,26775,267kWh Savings Error Margin0.06991250.0939295

    Number of Observations2,689,1922,689,1922,689,1922,689,192

    Notes: The dependent variable is customer electricity ADC for a month-year. All models are linear regression models. Standard errors in parentheses were clustered by customer. Cadmus estimated all models with pre-treatment and post-treatment monthly consumption data.

    Table 7. Natural Gas Regression Estimates

    Phase 1 (June 2015 – May 2016)Phase 2 (June 2016 – April 2018)

    Model 1Model 2Model 1Model 2Phase 1Phase 2

    Average Daily Savings per Customer (therms)0.00910.0090.00980.0097% Savings0.5%0.5%

    (0.004)(0.004)(0.004)(0.004)kWh Savings0.0090.0097

    Customer Fixed EffectsYesYesYesYesBaseline1.66666666672.0208333333

    Weather VariablesNoYesNoYes

    Month-Year Fixed EffectsYesYesYesYes% Savings Error Margin0.4%0.3%

    Number of Customers75,25675,25675,25675,256kWh Savings Error Margin0.0059220.0067445

    Number of Observations2,684,7072,684,7072,684,7072,684,707

    Notes: The dependent variable is customer natural gas ADC for a month-year. All models are linear regression models. Standard errors in parentheses were clustered by customer. Cadmus estimated all models with pre-treatment and post-treatment monthly consumption data.

    % Savings

    3.2686363636363638E-34.0656052238805963E-33.2686363636363638E-34.0656052238805963E-3Phase 1Phase 23.5999999999999999E-35.7999999999999996E-3

    kWh Savings

    [VALUE]

    6.9912500000000002E-29.3929499999999999E-26.9912500000000002E-29.3929499999999999E-2Phase 1Phase 27.6999999999999999E-20.13400000000000001

    Avg. Daily Savings per Customer

    % Savings

    3.5532000000000003E-33.3374845360824744E-33.5532000000000003E-33.3374845360824744E-3Phase 1Phase 25.4000000000000003E-34.7999999999999996E-3

    kWh Savings

    3.5532000000000003E-33.3374845360824744E-33.5532000000000003E-33.3374845360824744E-3Phase 1Phase 28.9999999999999993E-39.7000000000000003E-3

    Avg. Daily Savings per Customer

    Sheet2

    FuelNumber of Customers Sent HBAsNumber of HBAs SentAverage Number of HBAs per Treated Customer

    Electricity31,591100,3203.2

    Natural Gas6,16313,3552.2

    Fuel

    Number of Customers Sent High Bill Alerts

    Total Number of High Bill Alerts Sent

    Average Number of High Bill Alerts per Treatment Group Customer

    Electricity

    31,591

    100,320

    3.2

    Natural Gas

    6,163

    13,355

    2.2

    Fuel

    Number of Customers

    Sent High Bill Alerts

    Total Number of High Bill

    Alerts Sent

    Average Number of High Bill Alerts

    per Treatment Group Customer

    Electricity 31,591 100,320 3.2

    Natural Gas 6,163 13,355 2.2

  • IEPEC, 2019 Denver

    Research Objectives

    • No statistically significant effect on customer call center call volumes (Source: Opower/Oracle analysis)

    • Cadmus research objective • Estimate customer gas and electricity savings during and after the pilot

    6

  • IEPEC, 2019 Denver

    HBA Pilot Evaluation ApproachFindingsTakeaways and Future Research

    7

    Agenda

  • IEPEC, 2019 Denver

    Evaluation Approach

    • Pilot implemented as an RCT• 50,000 treatment group customers and 25,000 control group customers• Only treatment group customers eligible to receive HBAs

    • Monthly billing consumption data• Validation of randomized experiment• Fixed-effects D-in-D panel regression • Estimated savings for two phases:

    • Phase 1: HBAs delivered (June 2015 - June 2016) • Phase 2: HBA delivery suspended (July 2016 – April 2018)

    8

  • IEPEC, 2019 Denver

    HBA Pilot Evaluation ApproachFindingsTakeaways and Future Research

    9

    Agenda

  • IEPEC, 2019 Denver

    Electricity Savings Estimates

    10

    Percentage SavingskWh Savings

    Note: Error bars indicated 90% confidence intervals based on standard errors clustered on customers.

    • Electricity savings persisted after Xcel Energy stopped sending HBAs• In Phase 2, electricity savings per customer who received an HBA

    equaled 0.16 kWh (0.7%)

    0.4%

    0.6%

    0.0%

    0.2%

    0.4%

    0.6%

    0.8%

    1.0%

    1.2%

    Phase 1 Phase 2

    0.08

    0.13

    0.00

    0.05

    0.10

    0.15

    0.20

    0.25

    Phase 1 Phase 2

    Avg.

    Dai

    ly Sa

    ving

    s per

    Cus

    tom

    er

    Chart9

    % Savings

    3.6318181818181818E-34.2057985074626858E-33.6318181818181818E-34.2057985074626858E-3Phase 1Phase 24.0000000000000001E-36.0000000000000001E-3

    Sheet1

    Electricity Savings

    Phase 1 (June 2015 – May 2016)Phase 2 (June 2016 – April 2018)

    Model 1Model 2Model 1Model 2Phase 1Phase 2

    Average Daily Savings per Customer (kWh)0.0770.0770.1330.134% Savings0.4%0.6%

    (0.043)(0.043)(0.057)(0.057)kWh Savings0.0770.134

    Customer Fixed EffectsYesYesYesYesBaseline19.2522.3333333333

    Weather VariablesNoYesNoYes

    Month-Year Fixed EffectsYesYesYesYes% Savings Error Margin0.36%0.42%

    Number of Customers75,26775,26775,26775,267kWh Savings Error Margin0.06991250.0939295

    Number of Observations2,689,1922,689,1922,689,1922,689,192

    Notes: The dependent variable is customer electricity ADC for a month-year. All models are linear regression models. Standard errors in parentheses were clustered by customer. Cadmus estimated all models with pre-treatment and post-treatment monthly consumption data.

    Table 7. Natural Gas Regression Estimates

    Phase 1 (June 2015 – May 2016)Phase 2 (June 2016 – April 2018)

    Model 1Model 2Model 1Model 2Phase 1Phase 2

    Average Daily Savings per Customer (therms)0.00910.0090.00980.0097% Savings0.5%0.5%

    (0.004)(0.004)(0.004)(0.004)kWh Savings0.0090.0097

    Customer Fixed EffectsYesYesYesYesBaseline1.81.94

    Weather VariablesNoYesNoYes

    Month-Year Fixed EffectsYesYesYesYes% Savings Error Margin0.3%0.3%

    Number of Customers75,25675,25675,25675,256kWh Savings Error Margin0.0059220.0067445

    Number of Observations2,684,7072,684,7072,684,7072,684,707

    Notes: The dependent variable is customer natural gas ADC for a month-year. All models are linear regression models. Standard errors in parentheses were clustered by customer. Cadmus estimated all models with pre-treatment and post-treatment monthly consumption data.

    % Savings

    3.6318181818181818E-34.2057985074626858E-33.6318181818181818E-34.2057985074626858E-3Phase 1Phase 24.0000000000000001E-36.0000000000000001E-3

    kWh Savings

    [VALUE]

    6.9912500000000002E-29.3929499999999999E-26.9912500000000002E-29.3929499999999999E-2Phase 1Phase 27.6999999999999999E-20.13400000000000001

    Avg. Daily Savings per Customer

    % Savings

    3.2900000000000004E-33.4765463917525775E-33.2900000000000004E-33.4765463917525775E-3Phase 1Phase 25.0000000000000001E-35.0000000000000001E-3

    kWh Savings

    3.2900000000000004E-33.4765463917525775E-33.2900000000000004E-33.4765463917525775E-3Phase 1Phase 28.9999999999999993E-39.7000000000000003E-3

    Avg. Daily Savings per Customer

    Chart10

    kWh Savings

    [VALUE]

    6.9912500000000002E-29.3929499999999999E-26.9912500000000002E-29.3929499999999999E-2Phase 1Phase 27.6999999999999999E-20.13400000000000001

    Avg. Daily Savings per Customer

    Sheet1

    Electricity Savings

    Phase 1 (June 2015 – May 2016)Phase 2 (June 2016 – April 2018)

    Model 1Model 2Model 1Model 2Phase 1Phase 2

    Average Daily Savings per Customer (kWh)0.0770.0770.1330.134% Savings0.4%0.6%

    (0.043)(0.043)(0.057)(0.057)kWh Savings0.0770.134

    Customer Fixed EffectsYesYesYesYesBaseline19.2522.3333333333

    Weather VariablesNoYesNoYes

    Month-Year Fixed EffectsYesYesYesYes% Savings Error Margin0.36%0.42%

    Number of Customers75,26775,26775,26775,267kWh Savings Error Margin0.06991250.0939295

    Number of Observations2,689,1922,689,1922,689,1922,689,192

    Notes: The dependent variable is customer electricity ADC for a month-year. All models are linear regression models. Standard errors in parentheses were clustered by customer. Cadmus estimated all models with pre-treatment and post-treatment monthly consumption data.

    Table 7. Natural Gas Regression Estimates

    Phase 1 (June 2015 – May 2016)Phase 2 (June 2016 – April 2018)

    Model 1Model 2Model 1Model 2Phase 1Phase 2

    Average Daily Savings per Customer (therms)0.00910.0090.00980.0097% Savings0.5%0.5%

    (0.004)(0.004)(0.004)(0.004)kWh Savings0.0090.0097

    Customer Fixed EffectsYesYesYesYesBaseline1.81.94

    Weather VariablesNoYesNoYes

    Month-Year Fixed EffectsYesYesYesYes% Savings Error Margin0.3%0.3%

    Number of Customers75,25675,25675,25675,256kWh Savings Error Margin0.0059220.0067445

    Number of Observations2,684,7072,684,7072,684,7072,684,707

    Notes: The dependent variable is customer natural gas ADC for a month-year. All models are linear regression models. Standard errors in parentheses were clustered by customer. Cadmus estimated all models with pre-treatment and post-treatment monthly consumption data.

    % Savings

    3.6318181818181818E-34.2057985074626858E-33.6318181818181818E-34.2057985074626858E-3Phase 1Phase 24.0000000000000001E-36.0000000000000001E-3

    kWh Savings

    [VALUE]

    6.9912500000000002E-29.3929499999999999E-26.9912500000000002E-29.3929499999999999E-2Phase 1Phase 27.6999999999999999E-20.13400000000000001

    Avg. Daily Savings per Customer

    % Savings

    3.2900000000000004E-33.4765463917525775E-33.2900000000000004E-33.4765463917525775E-3Phase 1Phase 25.0000000000000001E-35.0000000000000001E-3

    kWh Savings

    3.2900000000000004E-33.4765463917525775E-33.2900000000000004E-33.4765463917525775E-3Phase 1Phase 28.9999999999999993E-39.7000000000000003E-3

    Avg. Daily Savings per Customer

  • IEPEC, 2019 Denver

    Monthly Electricity Savings Estimates

    11

    Note: Error bars indicate 90% confidence intervals based on standard errors clustered on customers.

    -0.2

    -0.1

    0

    0.1

    0.2

    0.3

    0.4

    0.5

    0.6

    Avg

    Dai

    ly S

    avin

    gs p

    er C

    usto

    mer

    (kW

    h)

    Estimated Savings 90% Lower Savings Estimate 90% Upper Savings Estimate

    Phase 1: HBA treatment period

    Chart1

    Estimated Savings4215642186422174224842278423094233942370424014243042461424914252242552425834261442644426754270542736427674279542826428564288742917429484297943009430404307043101431324316043191-4.9500000000000002E-21.67E-27.1400000000000005E-29.3799999999999994E-28.1000000000000003E-23.7199999999999997E-29.9500000000000005E-20.131000000000000010.115500000000000014.1099999999999998E-25.1400000000000001E-20.168199999999999990.1510.117000000000000010.108299999999999997.3599999999999999E-23.6799999999999999E-29.3799999999999994E-20.250099999999999990.231399999999999990.14097.2099999999999997E-28.6E-30.043.9600000000000003E-20.123799999999999990.143499999999999990.174499999999999998.0799999999999997E-20.16730.266500000000000010.295599999999999970.260699999999999990.123799999999999997.9200000000000007E-290% Lower Savings Estimate4215642186422174224842278423094233942370424014243042461424914252242552425834261442644426754270542736427674279542826428564288742917429484297943009430404307043101431324316043191-0.15494450000000001-0.11884800000000001-5.5429500000000007E-2-9.8350000000000104E-3-6.8430000000000019E-3-5.1959000000000005E-2-1.0715000000000002E-2-1.916000000000001E-3-5.5719999999999936E-3-6.0232000000000008E-2-4.4503500000000001E-26.6045499999999979E-21.2491000000000002E-2-5.029649999999998E-2-4.8632999999999996E-2-5.0268500000000008E-2-7.4073E-2-1.7566499999999999E-29.8430999999999991E-27.3644500000000002E-23.8714999999999999E-3-5.4893999999999998E-2-0.10753699999999999-7.7453000000000022E-2-0.10993049999999999-5.1886000000000015E-2-1.2610499999999997E-23.22075E-2-4.2904000000000012E-22.7968500000000007E-28.7853000000000014E-29.3593999999999955E-26.8234999999999962E-2-3.9548500000000014E-2-6.358599999999999E-290% Upper Savings Estimate42156421864221742248422784230942339423704240142430424614249142522425524258342614426444267542705427364276742795428264285642887429174294842979430094304043070431014313243160431915.5944500000000008E-20.152247999999999990.19822950.1974350.168843000000000020.1263590.209715000000000010.263916000000000040.2365720.1424320.14730350.27035450.289509000000000020.284296500000000010.2652330.197468499999999990.1476730.205166499999999970.401768999999999990.389155499999999990.277928500000000020.199093999999999990.124736999999999990.157453000000000010.189130499999999980.299486000000000030.29961050.316792499999999980.204504000000000020.30663150.445147000000000010.497605999999999990.453165000000000040.287148500000000030.22198600000000002

    Avg Daily Savings per Customer (kWh)

    Treatment Estimates

    util_typePeriodmodel_typemodelMOdel_nParmEstimateStderrProbZ

    Epost1DNDE_DND_SIMPLE1treatment_post1_norm-0.44840.04320.0001

    Epost1DNDE_DND_OVERALL2treatment_post1_norm-0.04730.04260.2678

    Epost1DNDE_DND_OVERALL_WEATHER3treatment_post1_norm-0.04730.04260.2674

    Epost1DNDE_DND_OVERALL_HDD4treatment_post1_norm-0.0470.04260.2704

    Epost1PO_E_PO_SIMPLE_NO_CTRL1treatment_post1-1.0990.1270.0001

    Epost1PO_E_PO_SIMPLE2treatment_post1-0.53950.04720.0001

    Epost1PO_E_PO_SIMPLE_PREINTERACTED3treatment_post1-0.31130.0490.0001

    Epost1PO_E_PO_AVGPRENOINT4treatment_post1-0.0130.04290.7624

    Epost1PO_E_PO_OVERALL5treatment_post1-0.00120.0450.978

    Epost1PO_E_PO_OVERALL_WEATHER6treatment_post1-0.0010.0450.9822

    Epost1PO_E_PO_OVERALL_HDD7treatment_post1-0.00120.0450.9793

    Epost2DNDE_DND_SIMPLE1treatment_post2_norm0.29140.04870.0001

    Epost2DNDE_DND_OVERALL2treatment_post2_norm-0.13190.05710.0209

    Epost2DNDE_DND_OVERALL_WEATHER3treatment_post2_norm-0.13240.05710.0203

    Epost2DNDE_DND_OVERALL_HDD4treatment_post2_norm-0.13170.05710.021

    Epost2PO_E_PO_SIMPLE_NO_CTRL1treatment_post20.39520.13030.0024

    Epost2PO_E_PO_SIMPLE2treatment_post20.10990.05330.0393

    Epost2PO_E_PO_SIMPLE_PREINTERACTED3treatment_post20.14960.05470.0062

    Epost2PO_E_PO_AVGPRENOINT4treatment_post2-0.15980.0620.01

    Epost2PO_E_PO_OVERALL5treatment_post2-0.14710.06290.0193

    Epost2PO_E_PO_OVERALL_WEATHER6treatment_post2-0.14730.06290.0192

    Epost2PO_E_PO_OVERALL_HDD7treatment_post2-0.14690.06290.0195

    Gpost1DNDG_DND_SIMPLE1treatment_post1_norm-0.16240.00430.0001

    Gpost1DNDG_DND_OVERALL2treatment_post1_norm-0.00920.00360.0106

    Gpost1DNDG_DND_OVERALL_WEATHER3treatment_post1_norm-0.00910.00360.0119

    Gpost1DNDG_DND_OVERALL_HDD4treatment_post1_norm-0.00910.00360.0113

    Gpost1PO_G_PO_SIMPLE_NO_CTRL1treatment_post1-0.15220.01110.0001

    Gpost1PO_G_PO_SIMPLE2treatment_post1-0.12280.01080.0001

    Gpost1PO_G_PO_SIMPLE_PREINTERACTED3treatment_post1-0.18220.0110.0001

    Gpost1PO_G_PO_AVGPRENOINT4treatment_post10.00530.00990.5961

    Gpost1PO_G_PO_OVERALL5treatment_post10.00430.010.6636

    Gpost1PO_G_PO_OVERALL_WEATHER6treatment_post10.00430.010.6641

    Gpost1PO_G_PO_OVERALL_HDD7treatment_post10.00440.010.6631

    Gpost2DNDG_DND_SIMPLE1treatment_post2_norm0.06170.00440.0001

    Gpost2DNDG_DND_OVERALL2treatment_post2_norm-0.00970.00410.0174

    Gpost2DNDG_DND_OVERALL_WEATHER3treatment_post2_norm-0.00960.00410.0187

    Gpost2DNDG_DND_OVERALL_HDD4treatment_post2_norm-0.00970.00410.0179

    Gpost2PO_G_PO_SIMPLE_NO_CTRL1treatment_post20.18410.01160.0001

    Gpost2PO_G_PO_SIMPLE2treatment_post20.16880.01130.0001

    Gpost2PO_G_PO_SIMPLE_PREINTERACTED3treatment_post20.13480.01150.0001

    Gpost2PO_G_PO_AVGPRENOINT4treatment_post20.00530.01210.665

    Gpost2PO_G_PO_OVERALL5treatment_post20.00510.01220.6733

    Gpost2PO_G_PO_OVERALL_WEATHER6treatment_post20.00510.01220.6743

    Gpost2PO_G_PO_OVERALL_HDD7treatment_post20.00520.01220.6714

    Elec Means

    E Pre ADC ttest both flags

    The TTEST Procedure

    Variable:  adc

    groupNMeanStd DevStd ErrMinimumMaximumgroupMethodMean90% CL MeanStd Dev90% CL Std Dev

    CONTROL2391120.180214.47630.09360378.3CONTROL20.180220.02620.33414.476314.368314.5861

    TREATMENT4793120.165913.66490.06240437.5GroupAverage Daily Electric Usage (kWh)Difference (Phase 1-Pre)Difference (Phase 2-Pre)TREATMENT20.165920.06320.26913.664913.592713.7379

    Diff (1-2)0.014313.94020.1104PrePhase 1Phase 2Diff (1-2)Pooled0.0143-0.16720.195913.940213.8814.001

    groupMethodMean90% CL MeanStd Dev90% CL Std DevControl20.18021.38823.0401.2072.860Diff (1-2)Satterthwaite0.0143-0.17080.1994

    CONTROL20.180220.026220.334214.476314.368314.5861Treatment20.16621.28322.8831.1172.717

    TREATMENT20.165920.063320.268613.664913.592713.7379Difference (T-C)-0.014-0.104-0.157-0.090-0.143

    Diff (1-2)Pooled0.0143-0.16720.195913.940213.8814.001

    Diff (1-2)Satterthwaite0.0143-0.17080.1994

    MethodVariancesDFt ValuePr > |t|

    PooledEqual718400.130.89680.6458333333

    SatterthwaiteUnequal454140.130.8988

    Equality of Variances

    MethodNum DFDen DFF ValuePr > F0.36%0.58%000

    Folded F23910479301.12 |t|

    PooledEqual680340.920.3583

    SatterthwaiteUnequal424690.90.3686

    Equality of Variances

    MethodNum DFDen DFF ValuePr > F

    Folded F22537454971.14 |t|

    PooledEqual537661.190.2336

    SatterthwaiteUnequal332731.160.2469

    Equality of Variances

    MethodNum DFDen DFF ValuePr > F

    Folded F17920358461.18 |t|

    PooledEqual58706-1.050.2956

    SatterthwaiteUnequal38315-1.040.3

    Equality of Variances

    MethodNum DFDen DFF ValuePr > F

    Folded F19606391001.06 |t|Diff (1-2)Satterthwaite-0.0141-0.03290.00478

    PooledEqual77623-1.230.2183

    SatterthwaiteUnequal51289-1.230.21970.54%0.48%

    Equality of Variances

    MethodNum DFDen DFF ValuePr > F

    Folded F25856517671.020.099

    G Post1 ADC ttest both flags

    The TTEST Procedure

    Variable:  adc

    groupNMeanStd DevStd ErrMinimumMaximum

    CONTROL247761.67741.24290.0079020.6695

    TREATMENT496491.67791.2130.00544020.6873

    Diff (1-2)-0.000541.2230.00951groupMethodMean90% CL MeanStd Dev90% CL Std Dev

    groupMethodMean90% CL MeanStd Dev90% CL Std DevCONTROL1.67741.66441.69041.24291.23381.2522

    CONTROL1.67741.66441.69041.24291.23381.2522TREATMENT1.67791.66891.68691.2131.20671.2194

    TREATMENT1.67791.66891.68691.2131.20671.2194Diff (1-2)Pooled-0.00054-0.01620.01511.2231.21781.2283

    Diff (1-2)Pooled-0.00054-0.01620.01511.2231.21781.2283Diff (1-2)Satterthwaite-0.00054-0.01630.0152

    Diff (1-2)Satterthwaite-0.00054-0.01630.0152

    MethodVariancesDFt ValuePr > |t|

    PooledEqual74423-0.060.9548

    SatterthwaiteUnequal48459-0.060.9552

    Equality of Variances

    MethodNum DFDen DFF ValuePr > F

    Folded F24775496481.05 |t|

    PooledEqual58706-1.050.2956

    SatterthwaiteUnequal38315-1.040.3

    Equality of Variances

    MethodNum DFDen DFF ValuePr > F

    Folded F19606391001.06 |Z|

    Error

    Intercept0000..

    treatment_post1_norm0.07730.0425-0.16070.0061-1.820.06940.00738750.1472125

    treatment_post2_norm0.13360.0571-0.2455-0.0218-2.340.01920.03967050.2275295

    yr2014_mo6_norm3.06640.10112.86823.264630.32

  • IEPEC, 2019 Denver

    Gas Savings

    12

    Percentage Savings

    0.0090.010

    0.000

    0.002

    0.004

    0.006

    0.008

    0.010

    0.012

    0.014

    Phase 1 Phase 2

    Avg.

    Dai

    ly S

    avin

    gs p

    er C

    usto

    mer

    0.5% 0.5%

    0.0%

    0.2%

    0.4%

    0.6%

    0.8%

    1.0%

    Phase 1 Phase 2

    Therm Savings

    Note: Error bars indicated 90% confidence intervals based on standard errors clustered on customers.

    • Gas savings persisted after Xcel Energy stopped sending HBAs• In Phase 2, gas savings per customer who received an HBA equaled

    0.011 therms (0.6%)

  • IEPEC, 2019 Denver

    Monthly Gas Savings Estimates

    13

    Note: Error bars indicate 90% confidence intervals based on standard errors clustered on customers.

    -0.04

    -0.03

    -0.02

    -0.01

    0

    0.01

    0.02

    0.03

    0.04

    0.05

    0.06

    Aver

    age

    Dai

    ly S

    avin

    gs p

    er C

    usto

    mer

    (the

    rms)

    Estimated Savings 90% Lower Savings Estimate 90% Upper Savings Estimate

    Phase 1: HBA treatment period

    Chart2

    Estimated Savings42156421864221742248422784230942339423704240142430424614249142522425524258342614426444267542705427364276742795428264285642887429174294842979430094304043070431014313243160431918.6999999999999994E-36.7999999999999996E-37.4999999999999997E-37.6E-31.26E-21.41E-21.06E-24.1000000000000003E-33.7000000000000002E-38.0999999999999996E-31.0999999999999999E-21.21E-27.4000000000000003E-36.0000000000000001E-31.8E-35.1999999999999998E-38.8999999999999999E-39.5999999999999992E-31.37E-21.2E-29.1000000000000004E-31.0800000000000001E-29.5999999999999992E-31.2800000000000001E-27.9000000000000008E-34.0000000000000001E-31.11E-21.52E-21.43E-21.4500000000000001E-21.61E-21.6500000000000001E-22.3E-31.17E-21.01E-290% Lower Savings Estimate4215642186422174224842278423094233942370424014243042461424914252242552425834261442644426754270542736427674279542826428564288742917429484297943009430404307043101431324316043191-6.4340000000000005E-3-8.992E-3-8.6210000000000002E-3-7.6984999999999996E-31.5785E-36.5329999999999997E-3-3.876000000000001E-3-1.90945E-2-1.3407999999999998E-2-4.5399999999999954E-42.7749999999999997E-3-1.0600000000000002E-3-9.049999999999999E-3-1.09435E-2-1.5307999999999999E-2-1.1085500000000002E-2-3.1085000000000002E-35.5249999999999917E-4-9.4944999999999995E-3-1.2017E-2-7.8434999999999998E-3-1.5374999999999989E-3-2.7000000000000114E-4-1.0179999999999981E-3-9.3724999999999989E-3-1.4095E-2-7.4884999999999969E-3-2.2370000000000011E-33.4429999999999999E-3-4.6949999999999943E-4-1.1207000000000002E-2-1.7715999999999996E-2-2.6322999999999999E-2-5.4079999999999979E-3-7.5700000000000073E-490% Upper Savings Estimate42156421864221742248422784230942339423704240142430424614249142522425524258342614426444267542705427364276742795428264285642887429174294842979430094304043070431014313243160431912.3834000000000001E-22.2592000000000001E-22.3621E-22.2898499999999999E-22.36215E-22.1666999999999999E-22.5076000000000001E-22.7294499999999999E-22.0808E-21.6653999999999999E-21.9224999999999999E-22.5259999999999998E-22.385E-22.2943499999999999E-21.8907999999999998E-22.1485500000000001E-22.09085E-21.8647499999999997E-23.6894499999999997E-23.6017E-22.6043500000000001E-22.3137499999999998E-21.9470000000000001E-22.6617999999999999E-22.51725E-22.2095E-22.96885E-23.2636999999999999E-22.5156999999999999E-22.9469500000000003E-24.3407000000000001E-25.0715999999999997E-23.0922999999999999E-22.8808E-22.0957E-2

    Average Daily Savings per Customer (therms)

    Treatment Estimates

    util_typePeriodmodel_typemodelMOdel_nParmEstimateStderrProbZ

    Epost1DNDE_DND_SIMPLE1treatment_post1_norm-0.44840.04320.0001

    Epost1DNDE_DND_OVERALL2treatment_post1_norm-0.04730.04260.2678

    Epost1DNDE_DND_OVERALL_WEATHER3treatment_post1_norm-0.04730.04260.2674

    Epost1DNDE_DND_OVERALL_HDD4treatment_post1_norm-0.0470.04260.2704

    Epost1PO_E_PO_SIMPLE_NO_CTRL1treatment_post1-1.0990.1270.0001

    Epost1PO_E_PO_SIMPLE2treatment_post1-0.53950.04720.0001

    Epost1PO_E_PO_SIMPLE_PREINTERACTED3treatment_post1-0.31130.0490.0001

    Epost1PO_E_PO_AVGPRENOINT4treatment_post1-0.0130.04290.7624

    Epost1PO_E_PO_OVERALL5treatment_post1-0.00120.0450.978

    Epost1PO_E_PO_OVERALL_WEATHER6treatment_post1-0.0010.0450.9822

    Epost1PO_E_PO_OVERALL_HDD7treatment_post1-0.00120.0450.9793

    Epost2DNDE_DND_SIMPLE1treatment_post2_norm0.29140.04870.0001

    Epost2DNDE_DND_OVERALL2treatment_post2_norm-0.13190.05710.0209

    Epost2DNDE_DND_OVERALL_WEATHER3treatment_post2_norm-0.13240.05710.0203

    Epost2DNDE_DND_OVERALL_HDD4treatment_post2_norm-0.13170.05710.021

    Epost2PO_E_PO_SIMPLE_NO_CTRL1treatment_post20.39520.13030.0024

    Epost2PO_E_PO_SIMPLE2treatment_post20.10990.05330.0393

    Epost2PO_E_PO_SIMPLE_PREINTERACTED3treatment_post20.14960.05470.0062

    Epost2PO_E_PO_AVGPRENOINT4treatment_post2-0.15980.0620.01

    Epost2PO_E_PO_OVERALL5treatment_post2-0.14710.06290.0193

    Epost2PO_E_PO_OVERALL_WEATHER6treatment_post2-0.14730.06290.0192

    Epost2PO_E_PO_OVERALL_HDD7treatment_post2-0.14690.06290.0195

    Gpost1DNDG_DND_SIMPLE1treatment_post1_norm-0.16240.00430.0001

    Gpost1DNDG_DND_OVERALL2treatment_post1_norm-0.00920.00360.0106

    Gpost1DNDG_DND_OVERALL_WEATHER3treatment_post1_norm-0.00910.00360.0119

    Gpost1DNDG_DND_OVERALL_HDD4treatment_post1_norm-0.00910.00360.0113

    Gpost1PO_G_PO_SIMPLE_NO_CTRL1treatment_post1-0.15220.01110.0001

    Gpost1PO_G_PO_SIMPLE2treatment_post1-0.12280.01080.0001

    Gpost1PO_G_PO_SIMPLE_PREINTERACTED3treatment_post1-0.18220.0110.0001

    Gpost1PO_G_PO_AVGPRENOINT4treatment_post10.00530.00990.5961

    Gpost1PO_G_PO_OVERALL5treatment_post10.00430.010.6636

    Gpost1PO_G_PO_OVERALL_WEATHER6treatment_post10.00430.010.6641

    Gpost1PO_G_PO_OVERALL_HDD7treatment_post10.00440.010.6631

    Gpost2DNDG_DND_SIMPLE1treatment_post2_norm0.06170.00440.0001

    Gpost2DNDG_DND_OVERALL2treatment_post2_norm-0.00970.00410.0174

    Gpost2DNDG_DND_OVERALL_WEATHER3treatment_post2_norm-0.00960.00410.0187

    Gpost2DNDG_DND_OVERALL_HDD4treatment_post2_norm-0.00970.00410.0179

    Gpost2PO_G_PO_SIMPLE_NO_CTRL1treatment_post20.18410.01160.0001

    Gpost2PO_G_PO_SIMPLE2treatment_post20.16880.01130.0001

    Gpost2PO_G_PO_SIMPLE_PREINTERACTED3treatment_post20.13480.01150.0001

    Gpost2PO_G_PO_AVGPRENOINT4treatment_post20.00530.01210.665

    Gpost2PO_G_PO_OVERALL5treatment_post20.00510.01220.6733

    Gpost2PO_G_PO_OVERALL_WEATHER6treatment_post20.00510.01220.6743

    Gpost2PO_G_PO_OVERALL_HDD7treatment_post20.00520.01220.6714

    Elec Means

    E Pre ADC ttest both flags

    The TTEST Procedure

    Variable:  adc

    groupNMeanStd DevStd ErrMinimumMaximumgroupMethodMean90% CL MeanStd Dev90% CL Std Dev

    CONTROL2391120.180214.47630.09360378.3CONTROL20.180220.02620.33414.476314.368314.5861

    TREATMENT4793120.165913.66490.06240437.5GroupAverage Daily Electric Usage (kWh)Difference (Phase 1-Pre)Difference (Phase 2-Pre)TREATMENT20.165920.06320.26913.664913.592713.7379

    Diff (1-2)0.014313.94020.1104PrePhase 1Phase 2Diff (1-2)Pooled0.0143-0.16720.195913.940213.8814.001

    groupMethodMean90% CL MeanStd Dev90% CL Std DevControl20.18021.38823.0401.2072.860Diff (1-2)Satterthwaite0.0143-0.17080.1994

    CONTROL20.180220.026220.334214.476314.368314.5861Treatment20.16621.28322.8831.1172.717

    TREATMENT20.165920.063320.268613.664913.592713.7379Difference (T-C)-0.014-0.104-0.157-0.090-0.143

    Diff (1-2)Pooled0.0143-0.16720.195913.940213.8814.001

    Diff (1-2)Satterthwaite0.0143-0.17080.1994

    MethodVariancesDFt ValuePr > |t|

    PooledEqual718400.130.89680.6458333333

    SatterthwaiteUnequal454140.130.8988

    Equality of Variances

    MethodNum DFDen DFF ValuePr > F0.36%0.58%000

    Folded F23910479301.12 |t|

    PooledEqual680340.920.3583

    SatterthwaiteUnequal424690.90.3686

    Equality of Variances

    MethodNum DFDen DFF ValuePr > F

    Folded F22537454971.14 |t|

    PooledEqual537661.190.2336

    SatterthwaiteUnequal332731.160.2469

    Equality of Variances

    MethodNum DFDen DFF ValuePr > F

    Folded F17920358461.18 |t|

    PooledEqual58706-1.050.2956

    SatterthwaiteUnequal38315-1.040.3

    Equality of Variances

    MethodNum DFDen DFF ValuePr > F

    Folded F19606391001.06 |t|Diff (1-2)Satterthwaite-0.0141-0.03290.00478

    PooledEqual77623-1.230.2183

    SatterthwaiteUnequal51289-1.230.21970.54%0.48%

    Equality of Variances

    MethodNum DFDen DFF ValuePr > F

    Folded F25856517671.020.099

    G Post1 ADC ttest both flags

    The TTEST Procedure

    Variable:  adc

    groupNMeanStd DevStd ErrMinimumMaximum

    CONTROL247761.67741.24290.0079020.6695

    TREATMENT496491.67791.2130.00544020.6873

    Diff (1-2)-0.000541.2230.00951groupMethodMean90% CL MeanStd Dev90% CL Std Dev

    groupMethodMean90% CL MeanStd Dev90% CL Std DevCONTROL1.67741.66441.69041.24291.23381.2522

    CONTROL1.67741.66441.69041.24291.23381.2522TREATMENT1.67791.66891.68691.2131.20671.2194

    TREATMENT1.67791.66891.68691.2131.20671.2194Diff (1-2)Pooled-0.00054-0.01620.01511.2231.21781.2283

    Diff (1-2)Pooled-0.00054-0.01620.01511.2231.21781.2283Diff (1-2)Satterthwaite-0.00054-0.01630.0152

    Diff (1-2)Satterthwaite-0.00054-0.01630.0152

    MethodVariancesDFt ValuePr > |t|

    PooledEqual74423-0.060.9548

    SatterthwaiteUnequal48459-0.060.9552

    Equality of Variances

    MethodNum DFDen DFF ValuePr > F

    Folded F24775496481.05 |t|

    PooledEqual58706-1.050.2956

    SatterthwaiteUnequal38315-1.040.3

    Equality of Variances

    MethodNum DFDen DFF ValuePr > F

    Folded F19606391001.06 |Z|

    Error

    Intercept0000..

    treatment_post1_norm0.07730.0425-0.16070.0061-1.820.06940.00738750.1472125

    treatment_post2_norm0.13360.0571-0.2455-0.0218-2.340.01920.03967050.2275295

    yr2014_mo6_norm3.06640.10112.86823.264630.32

  • IEPEC, 2019 Denver

    HBA Pilot Evaluation ApproachFindingsTakeaways and Future Research

    14

    Agenda

  • IEPEC, 2019 Denver

    Takeaways

    15

    #1: HBAs did not reduce call center call volumes but generated energy savings- Electricity and natural gas savings of about 0.5% during treatment

    #2: Gas and electricity savings persisted after HBA treatment ended- Electricity savings increased after Xcel Energy stopped sending HBAs

    #3: HBAs have other potential benefits that this evaluation did not assess- Utility customer welfare

    • Customers are inattentive• HBAs may reduce costs of tracking consumption, and help customers avoid

    billing surprises, regret, and billing arrearages- Customer satisfaction with the utility

  • IEPEC, 2019 Denver

    Outstanding Research Questions

    16

    #1: What are effects of HBAs on customer engagement, satisfaction, and welfare?- Value from alerts- Behavioral changes- Impact on customer satisfaction with utility

    Research approach: Customer surveys, willingness-to-pay analysis

    #2: How do HBAs generate energy savings? Why do HBA savings persist?- Visits to Home Energy Assessment website?- Participation in Excel Energy efficiency programs?

    Research approach: Channeling/joint savings analysis

  • IEPEC, 2019 Denver

    Authors

    17

    Ryan FullemanSenior AnalystCadmus Energy [email protected]

    Julie HermanProduct DeveloperXcel [email protected]

    Grant Jacobsen, Ph.D.Senior Economist, Cadmus Energy ServicesAssociate Professor, University of [email protected]

    Jim Stewart, Ph.D.Principal EconomistCadmus Energy [email protected]

  • IEPEC, 2019 Denver18

    Thank You

    Questions?

    Slide Number 1AgendaSlide Number 3Slide Number 4Slide Number 5Slide Number 6AgendaSlide Number 8AgendaSlide Number 10Slide Number 11Slide Number 12Slide Number 13AgendaSlide Number 15Slide Number 16Slide Number 17Slide Number 18