California Public Utilities Commission Energy Efficiency Program Evaluation … Group B... ·...

California Public Utilities Commission

Energy Efficiency Program Evaluation of the Group B Sectors

Research Workplan

WE&T Sector Area

December 3, 2019

opiniondynamics.com Page ii

With Subcontractors

opiniondynamics.com Page iii

Table of Contents

1. Workforce Education & Training (WE&T) .......................................................................................................... 1

1.1 Area Staffing Plan .................................................................................................................................... 3

1.2 Deliverable 26 (WE&T-1): WE&T and Installation Improvement Evaluation Study ............................. 4

1.3 Deliverable 27 (WE&T-2): Partnerships with Training Institutions Impact Evaluation ...................... 15

1.4 Deliverable 28 (WE&T-3): WE&T Career Connections Process Evaluation ........................................ 23

1.5 Deliverable 29 (WE&T-4): Career and Workforce Readiness Process Evaluation ............................ 28

1.6 Deliverable 30 (WE&T-5): Knowledge, Skills, and Abilities Market Studies ...................................... 32

Workforce Education & Training (WE&T)

opiniondynamics.com Page 1

1. Workforce Education & Training (WE&T)

The California Long Term Energy Efficiency Strategic Plan (CLTEESP) sets forth the WE&T vision, “By 2020,

California’s workforce is trained and fully engaged to provide the human capital necessary to achieve

California’s economic energy efficiency and demand-side management potential.”1 This document outlines

the work scopes for the CPUC Group B WE&T research and evaluation studies, which together examine

multiple facets of the Workforce Education and Training landscape. Through these deliverables, we will:

◼ Assess the causal link between IOU workforce training programs and improved installation practices

(Deliverable 26);

◼ Seek to understand the functioning and impacts of partnerships with WE&T training and job

placement organizations (Deliverable 27);

◼ Conduct a process evaluation of the WE&T Career Connections Program (Deliverable 28);

◼ Conduct a process evaluation of the Career & Workforce Readiness Programs (Deliverable 29); and,

◼ Identify knowledge, skills, and abilities (KSA) needed for implementers, contractors, technicians, or

contractor laborers on energy efficiency concepts (Deliverable 30).

Conceptual Models

This research will be guided by two complementary conceptual frameworks: Bloom’s Taxonomy for the

Cognitive Domain and Kirkpatrick’s

Model for training evaluation. Bloom’s

Taxonomy is useful for defining

learning objectives and learning

outcomes while Kirkpatrick’s model

is broader in focus and addresses

the overall assessment of training

programs.

Bloom’s Cognitive Taxonomy is a

core element in the education and

training community. The full

taxonomy addresses three

domains: Cognitive (thinking),

Affective (feeling), and Psychomotor

(physical movement.) However, the

Cognitive Domain is the most

comprehensive. The original

taxonomy, established by Bloom in 1956, defined the levels in the cognitive domain as Knowledge,

Comprehension, Application, Analysis, Synthesis, and Evaluation. These levels were refined in the 1990s to

reflect the levels shown in Error! Reference source not found..

1 California Public Utilities Commission. “California Long Term Energy Efficiency Strategic Plan” Sept. 2008, updated Jan. 2011.

Accessed May 8, 2017. http://www.leginfo.ca.gov/pub/15-16/bill/sen/sb_0301-0350/sb_350_bill_20150911_enrolled.pdf

Figure 1: Bloom's Updated Cognitive Taxonomy



The Kirkpatrick Model is the gold standard framework in adult training circles for assessing training programs

and is noted in The California Evaluation Framework discussion of Information and Education programs.2

Where applicable, we will utilize this framework to guide our approach to these five WE&T studies. As

illustrated in Figure 2, Kirkpatrick’s Framework consists of four levels. The first level is Reaction. Reaction

measures how participants feel

about the learning experience. The

value of Level 1 is that a good

training experience improves

knowledge transfer. Level 2 is

Learning. Learning assesses the

degree participants change

attitudes, increase knowledge, or

enhance skills as a result of the

learning experience. The value of

Level 2 is to demonstrate that

learning occurs as a result of the

training. The third level is Behavior.

Behavior measures the degree to

which participants apply what they

have learned outside of the learning

environment. This level seeks to

demonstrate whether trainees take

the information they learn and apply

it. Finally, Level 4 is Results. Results refer to the degree targeted outcomes are achieved system wide. For

results, we seek to measure the program’s overall impacts and tangible results, such as energy savings, job

creation, job placement, improved quality, and increased productivity. The value of measuring Level 4 is to

inform the return on training investment that a program, entity, or organization realizes from the training

endeavor.

Given the breadth of the work conducted under the WE&T program implementation budget and the limitations

in evaluation funding, one of the key areas of consideration is how to prioritize any evaluation efforts--

specifically when the focus is on assessing energy impacts in training interventions that have a large breadth

and depth of types, topics, pre- and post-knowledge level of attendee population, large variability in decision-

making ability of the attendee population, participation population type and so forth. With this in mind, Opinion

Dynamics has worked to balance level of rigor, timeframes, potential data availability and scope among the

five WE&T studies. However, we are flexible and adaptable to changes in the work scopes based on what we

learn each year compiling Deliverable 06: Gaps and Emerging Issues Report and through conversations with

the Energy Division Project Manager for WE&T and the Project Coordination Group (PCG).

2 Note that Bloom’s Cognitive Domain Taxonomy and Kirkpatrick’s Model are complementary. The first is more focused on learning

objective and training design and the latter is a broader framework centered on evaluation.

Figure 2: Kirkpatrick’s Model



1.1 Area Staffing Plan

Dr. Ellen Steiner, Vice President at Opinion Dynamics, will serve as the overall area lead for the WE&T

deliverables. Dr. Steiner is a passionate researcher who embraces analytic challenges and is committed to

ensuring the useful application of results and insights to inform systems thinking and strategy development.

Dr. Steiner has 19 years of experience in training and development. She served as a professor in the

Department of Education at the University of Denver teaching graduate level research and statistic courses,

as well as educational assessment courses focusing on the K-12 population. She also advised over 100

doctoral dissertations. Dr. Steiner parlayed this experience into work at Sun Microsystems where she designed

the company’s corporate-wide competency management system--conducting pre-inventory data reviews, job-

task analyses, and verification processes to define knowledge, skills and abilities for a wide range of job

families. She also created supporting learning paths to develop the organization’s current and future skill

base. Dr. Steiner has evaluated numerous training programs – not just in energy, but also for Sun

Microsystems, and the University of Denver. She holds a Certificate in Measuring and Evaluation of Training

Programs from the American Society for Training and Development. In addition to her adult education and

training expertise, Dr. Steiner’s specialties include qualitative research, program evaluation, policy analysis,

ZNE, HVAC, customer engagement, market research, and mixed-method research designs.

Dr. Steiner holds a Ph.D. in Quantitative Research Methods from the University of Denver, an M.A. in Higher

Education Administration from the University of Denver, and a B.S. in Human Development and Family Studies

from the University of Delaware. Dr. Steiner’s background in social sciences and her applied business

experience support her effective communication and application of results and insights to inform cutting edge

program design, program evaluation, customer experience initiatives, and strategy development.

Deliverable Timelines

In Table 1, we identify the timelines for Deliverables 26-30.

Table 1. Consolidated Timelines for WE&T Studies

Task Q4

2018

Q1

2019

Q2

2019

Q3

2019

Q4

2019

Q1

2020

Q2

2020

Q3

2020

Q4

2020

Q1

2021

Q2

2021

Q3

2021

Q4

2021

Final Workplans x x

Deliverable 26

(WE&T-1): WE&T

and Installation

Improvement

Evaluation Study

x x x x x x x x x x x x

Deliverable 27

(WE&T-2):

Partnerships with

Training Institutions

Impact Evaluation

x x x x x x x x x x x

Deliverable 28

(WE&T-3): WE&T

Career Connections

Process Evaluation

x x x x x

Deliverable 29

(WE&T-4): Career x x x x x



Task Q4

2018

Q1

2019

Q2

2019

Q3

2019

Q4

2019

Q1

2020

Q2

2020

Q3

2020

Q4

2020

Q1

2021

Q2

2021

Q3

2021

Q4

2021

and Workforce

Readiness Process

Evaluation

Deliverable 30

(WE&T-5):

Knowledge, Skills,

and Abilities Market

Studies

x x x x x

1.2 Deliverable 26 (WE&T-1): WE&T and Installation Improvement

Evaluation Study

Past research3 has demonstrated that market actors who participate in education and training programs make

changes to their practices that result in energy savings. Over half (59%) of market actors who took Energy

Center courses between 2006 and 2009 made changes that became standard practice. Two of five market

actors (43%) said they made changes that resulted in self-described “measurable” energy savings.

The same study conducted a savings analysis of 29 market actors who took courses in lighting, building

envelope, and HVAC. The case studies estimated that these 29 market actors made changes based on what

they learned that resulted in approximately 10,000 MWh and a large majority of the market actors (87%)

shared what they learned with others. Thus, there is likely a multiplier effect where the course impacts extend

beyond the course taker and makes the potential for savings quite large.

This study is especially important to truly understand the causal links between installer training and installer

practice. For example, quality HVAC installation is a highly technical activity in which improper execution of the

necessary steps can lead to incorrect diagnoses of problems as well as solutions that can decrease efficiency;

and thus, it is essential installers are learning how to install HVAC units to maximize energy savings. However,

there are challenges attributing savings to market actors. Some of these challenges are at least partially

mitigated through the focus on installer practices in this particular study; thus limiting--to some extent--the

number of potential outcomes due to training. However, there are also challenges specific to market actors

such as:

◼ Attribution: Number and type of classes, level of knowledge of market actors, multiple classes by

market actors, multiplier effects create attribution issues

◼ Market actors likely apply what they learn numerous times across a variety of projects. Some

market actors may be able to describe a “typical project” that will accurately reflect most of their

work and could thus be extrapolated to all of their work in a given year. Other market actors work

on a variety projects and there is no typical project. As a result, it is difficult to conduct a survey

that is not overly burdensome yet also captures the full range of savings achieved.

3 Opinion Dynamics, Wirtshafter Associates, Inc., Jai Mitchell Analytics, Summit Blue Consulting, “Indirect Impact Evaluation of the

Statewide Energy Efficiency Education and Training Program.” Retrieved May 5, 2018.

http://www.calmac.org/publications/06%2D08%5FStatewide%5FEducation%5Fand%5FTraining%5FImpact%5FEval%5FVol%5FI%5F

FINAL%2Epdf



◼ Some market actors will work in areas that are not part of an IOU territory. It could be difficult to

ensure that savings is only estimated for those projects that took place in IOU territory.

◼ Market actors work on a number of projects and many may have received support from another

IOU program. It is challenging to avoid double counting of savings in these cases.

◼ Market actors are often one member within a team in new construction or retrofit practices, thus

influences over ultimate energy saving action may have had multiple sources.

◼ Past research has shown that market actors change their marketing practices as a result of what

they learn from the Energy Centers. The courses provide information they can use to sell customers

on more energy efficient options that the market actor may be aware of before taking the course.

It is challenging to estimate the additional installations and savings resulting from improved

marketing.

◼ These challenges are then compounded among market actors in that market actors can touch

multiple buildings that may vary in equipment installed, size of building, and usage patterns.

Moreover, education and training can not only affect the type of equipment installed, but also the

practices of the market actors.

◼ Data collection: Given the multitude of reach of market actors, and the potentially multiple influencing

factor over the construction and/or retrofit, data collection for this population is more complex, and

even in the eventuality that it can be collected, the actual energy savings estimate will require

assumptions on multiplier effects and attribution of savings.

Despite these challenges, there is an expectation that market actors in fact provide outsized savings,

especially when compared to end-users, given their potential reach. In addition, in the state of California, 94

PA energy efficiency programs in 2017 involved a contractor that performs retrofit installation and/or

maintenance work on existing buildings. Lighting and HVAC equipment are the most common equipment types

included in these programs--with 51% of programs involving one equipment type, the other, or both. Other

common measures include water heaters (15%), weatherization (14%), and refrigeration (12%). In addition,

most programs involve installation only (70%) or both installation and maintenance work (21%).4 Few

programs include maintenance only. Thus, installation is a high impact area for many of California’s key energy

efficiency programs making assessing energy savings for training of installers essential.

WE&T Program Administrators continue to develop classes that are geared toward market actors because of

the belief that they have a wider reach, and that they can have a trickle-down--and multiplier effect--across the

teams they work with as well as the sheer number of programs market actors touch. Thus, despite its

challenges, it is important to ascertain causal links between installer training and installer practice.

1.2.1 Study Objectives

The study objectives include:

◼ Assess the effectiveness of WE&T programs specifically among installation professionals;

◼ Assess the impact of each training workshop/session provided for relevant professions;

4 Opinion Dynamics, “Responsible Contractor Policy for EE Programs: Market Intelligence Study.” Accessed December 31, 2017.

http://www.calmac.org/publications/Responsible%5FContractor%5FPolicy%5FStudy%5FReport%5FFINAL%2Epdf



◼ Identify and assess the connection between training and improved installation practices;

◼ Provide recommendations on how to improve WE&T programs; and,

◼ Collect data to assess whether and to what extent WE&T programs drive energy savings.

Due to the broad scope of WE&T programs, and the wide range of energy applications and diverse set of trade

and professional job opportunities to which they apply, we will focus our investigation on residential and small

commercial HVAC (including heat pump water heaters) WE&T, exploring past data from the Residential Quality

Installation program to guide our assessment.

1.2.2 Overview of Evaluation Methodology

To assess the effectiveness of installer training, the Opinion Dynamics Team needs to determine the key

knowledge, skills and abilities (KSAs) that are needed to effectively conduct energy efficient installations.

Since the focus of Deliverable 30 is defining KSA’s for HVAC installation, we will focus this evaluation on HVAC

installation related courses. The overall objective of this study is to assess if there is a causal link between

IOU workforce training programs and improved installation practices, we have devoted our research resources

to focusing on examining this link. We will prepare a white paper investigating how NMEC-based analyses can

be employed to provide quantitative indicators of changes in installation practices as a result of training efforts

consistent with Kirkpatrick’s Level 4 criteria.

We will utilize the Kirkpatrick Model and Bloom’s Cognitive Taxonomy discussed above as frameworks of this

study. The Opinion Dynamics Team will also look for synergies between this study and Deliverable 27 as we

know several of the installer learning interventions are delivered through partnerships. The table below

identifies the methods we will utilize to meet the study objectives.

Table 2. WE&T Deliverable 26 Study Methodology by Research Objectives

Study Objective

Materials

&

Database

Review

Energy

Division

and

WE&T

Staff

Interviews

NMEC

Assessment

Exit

Survey

Installer

Surveys

Instructional

Design

Assessment

Assess

Learning

&

Behavior

Conduct

Impact

Evaluation

Assess the

effectiveness of

WE&T programs

specifically

among

installation

professionals

✓ ✓ ✓ ✓ ✓ ✓

Assess the impact

of each training

workshop/session

provided

✓ ✓ ✓ ✓ ✓ ✓ ✓

Identify and

assess the

connection

between training

✓ ✓ ✓ ✓ ✓ ✓



Study Objective

Materials

&

Database

Review

Energy

Division

and

WE&T

Staff

Interviews

NMEC

Assessment

Exit

Survey

Installer

Surveys

Instructional

Design

Assessment

Assess

Learning

&

Behavior

Conduct

Impact

Evaluation

and improved

installation

practices

Provide

recommendations

on how to

improve WE&T

programs

✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓

Collect data to

assess whether

and to what

extent WE&T

programs drive

energy savings.

✓ ✓ ✓ ✓ ✓ ✓

1.2.3 Evaluation Methodology: Detailed Efforts and Deliverables

Task 1. Materials and Database Review

Opinion Dynamics will complete a review of program materials, such as course catalogs, syllabi and other

available course materials, implementation plans, learning intervention types, exit survey data from courses,

and the course databases, program administrator business plans, past evaluations, key overarching studies

such as the Needs Assessment, and relevant policies, such as SB-350. We will utilize much of the information

obtained in Deliverable 30 to maximize research investment. We will submit an initial request for relevant

program materials, budgets and participant data available. Upon initial review of these data, we will submit a

second data request for specific information tied to courses focused on HVAC installers. These documents

may also include updated indictors, metrics, goals, participant tracking databases to date, and all available

curriculum plans.

Reviewing these materials will allow the team to:

◼ Develop a base understanding of the current program design and implementation including any sub-

components and how they address installers;

◼ Understand what data is available at the learning intervention level;

◼ Catalog the learning interventions targeted at installers in HVAC;

◼ Identify key program management and implementation staff we want to interview in Task 2 and

develop the questions that we want to pose to them; and,

◼ Ascertain what data is captured from exit surveys and how they meet Kirkpatrick’s Level 1

objectives.

Deliverables: Two Data requests



Task 2: Energy Division and WE&T Program Staff Interviews

The Opinion Dynamics Team will complete interviews with key individuals from the Energy Division, the PAs,

and stakeholders to develop a picture of how WE&T learning interventions are addressing the needs of HVAC

installers. The budget currently anticipates that we will conduct up to six interviews in 2019 and six interviews

in 2020.

The interviews with the Energy Division will be held in person or via telephone. All key Commission staff will be

invited to attend. Our key research team will conduct the Energy Division interviews.

PA staff will include the Center Directors and other key utility staff as determined to be relevant. These

interviews will take place via telephone. Through these interviews, we will:

◼ Discuss what changes are planned as a result of the business and implementation plans;

◼ Understand origin of metrics, indictors and goals, and the data to support them;

◼ Explore the effectiveness of various program components;

◼ Understand partnerships and the tracking of partnerships related to installation (if this data is not

already collected for Deliverable 27); and,

◼ Explore how adult learning principles are used/applied for each learning intervention targeted at

installers.

The outcomes of the interviews will consist of a complete and up-to-date set of current and future learning

interventions targeted at the installers of interest, a list of partners involved with the target audience, and

understanding of updated goals, indicators and metrics.

Deliverables: Draft and final interview guides

Task 3. Assessing the Use of NMEC Methodology to Evaluate WE&T Programs

In this task, Opinion Dynamics and Tierra Resource Consultants will develop a whitepaper that explores how

the overarching NMEC methodology may be applied to the evaluation of WE&T programs. In this whitepaper,

we will explore:

◼ What WE&T non-resource program performance metrics might be assessed through NMEC

techniques, including new metrics that will advance the use of NMEC?

◼ How program and meter data be can matched to participants in a WE&T non-resource activity such

that an NMEC analysis would provide meaningful results including quantitative metrics on changes in

installation practices?

◼ What is state of NMEC development and how might its evolving application relate to the WE&T sector?

◼ What is the most effective ways to develop baselines in a complex environment?

◼ Can NMEC methods be used to supplement data resources and provide new tools to specifically

assess the effectiveness of WE&T programs targeted at end users and market actors?



◼ Can NMEC methods be used to provide quantitative Kirkpatrick Level 4 performance metrics indicating

real changes in installation practices leading to energy savings?

Deliverables: Draft and final whitepaper

Task 4: Review and Refine Learning Intervention Exit Survey

The Opinion Dynamics Team will collaborate with the PA’s to make revisions to ensure those surveys covers

all aspects of Kirkpatrick’s level 1 evaluation. We will make sure the exit survey includes rating scale questions

and open-ended questions. Examples of such questions include:

◼ On a scale from 0 representing Strongly Disagree and 10 representing Strongly Agree, I was given

adequate opportunity to practice what I have learned.

◼ On a scale from 0 representing Not at All Confident and 10 representing Extremely Confident, how

confident are you that you will be able to apply what you have learned back on the job?

◼ What assistance or resources will you need to successfully apply what you learned on the job?

We will ask the utilities to use the same refined exit survey in all learning interventions targeted at installers

in 2020 and part of 2021. We will analyze this data to include in our final report

Deliverables: Refined exit survey, analysis of exit survey data

Task 5: Conduct Installer Surveys

Building upon any existing exit survey data, we will conduct surveys of HVAC installers who have participated

in a WE&T learning intervention in 2019. The objective of this survey is to address Kirkpatrick’s Levels 1-

Reaction, Level 2-Learning and Level 3-Behavior. Since we will likely have exit survey data that covers much

of Level 1, we will only include missing Level 1 components. We will focus on understanding:

◼ Reaction—Often called a “smile sheet”, we will assess satisfaction with all elements of a course.

◼ Learning—We will assess development of knowledge and skills as well as motivations to participate

and how they will use the learning experience to further their careers.

◼ Behavior—We will determine how participants have applied the knowledge and skills they learned in

the work place as well as identify any challenges they have encountered.

We will also focus on understanding the number of WE&T learning interventions installers have participated

in as well as key demographics. Finally, we will collect data that could be used to assess whether and to what

extent WE&T programs drive energy savings.

In order to determine the sample frame, we will utilize the program tracking database. In the Indirect Impact

Evaluation of the Statewide Energy Efficiency Education and Training Program Study,5 Opinion Dynamics et al.

made the recommendation that all Centers create a “common registration form that is used across all Centers

including participant type, profession, years in profession, and existing knowledge.” Unfortunately, this

5 Opinion Dynamics, Wirtshafter Associates, Inc., Jai Mitchell Analytics, Summit Blue Consulting, “Indirect Impact Evaluation of the

Statewide Energy Efficiency Education and Training Program.” Retrieved May 5, 2018.

http://www.calmac.org/publications/06%2D08%5FStatewide%5FEducation%5Fand%5FTraining%5FImpact%5FEval%5FVol%5FI%5F

FINAL%2Epdf



recommendation was not acted upon, so missing data will likely be an issue. For budgeting purposes, we have

planned to conduct an online survey with up to 200 installers. We will plan to refine our numbers based on

the data gathered in Tasks 1 and 2. We will offer a $75 gas card as a token of appreciation. From past

research, we know that gas cards are particularly motivating for this population. We have also budgeted to do

phone-call follow-ups to increase our response rate. We will conduct these surveys in Year 2.

Deliverables: Sample plan; draft and final survey

Task 6: Instructional Design Assessment

Using data from Task 1 and the survey in Task 5, Opinion Dynamics will perform an instructional design

assessment of up to 8 HVAC installer targeted WE&T learning interventions. We will gather data as described

below to address the following questions for each learning intervention.

◼ Do the WE&T learning interventions meet the needs of installers?

◼ Do the learning interventions target an appropriate range of learning levels and performance

outcomes (e.g. from basic learning levels, such as Knowledge and Comprehension, to more

sophisticated levels, such as Application, Analysis, Synthesis, and Evaluation6)?

◼ Do the objectives, content, and activities of the learning interventions target the knowledge, skills

and abilities defined in Deliverable 30?

◼ Do course materials and related supporting materials (e.g. checklists, job aids, and references)

help ensure an effective transfer from the classroom to the job? To the end-user?

◼ Are the learning interventions using effective learning strategies for the target audience?

◼ Do the materials reflect strategies and tactics that support adult learning principles (e.g. obtain

learner buy-in, build on what learners know, engage the learners, set up learners for success, let

learners apply what they have learned, etc.)?

◼ Do the materials reflect strategies and tactics consistent with best practices in adult learning (e.g.

lesson plan, content decisions, learner centricity, interactive activities, practice opportunities,

facilitation, feedback, assessments)?

◼ Are instructors delivering the training as it is designed — and if not, how do deliveries differ from

the apparent design intent?

◼ Do instructors employ adult learning principles and best practices effectively?

◼ How can WE&T programs improve offerings for installers?

◼ How can course focus, design, and materials better reflect the needs of the target audiences?

◼ How can the courses better employ learning strategies appropriate to class participants?

◼ How can instructors’ deliveries be refined to better incorporate the key adult learning principles

and practices?

6 Based on Bloom’s Updated Cognitive Taxonomy



In order to assess these learning interventions, we will conduct the following two sub-tasks for the up to 8

selected courses: (1) refine evaluation yardsticks; and, (2) evaluate course materials. These are described

below.

Refine Evaluation Yardsticks

The Opinion Dynamics Team will refine the evaluation yardsticks that we developed in past process

evaluations for California WE&T that focus on adult learning principles and practices, as well as program goals

and objectives. We will incorporate the HVAC KSA’s developed in Deliverable 30 and update the program goals

and objectives targeting installers. Whenever appropriate, we will use the same criteria used in the previous

process evaluations, so we can effectively compare results from those efforts to findings from this study. At

this time, we anticipate that it will address how well the training:

◼ Employs adult learning best practices;

◼ Supports installer-specific needs; and,

◼ Develops job-relevant knowledge, skills and abilities (helps prepare participants for job advancement

or job placement).

Our team will conduct a “yardstick review” of relevant available HVAC installer course materials for learning

interventions targeted at installers. We anticipate that the available materials will vary among the courses,

and may include:

◼ Lesson plans

◼ Instructor guides

◼ Presentation materials

◼ Participant workbooks

◼ References, job aids, and other participant support materials

◼ Materials for demonstrations or hands-on practice

◼ “Homework” assignments, if the course spans multiple days or is tightly integrated with other courses

◼ Quizzes, tests, polls, etc.

Based on our work thus far, we will coordinate with the Energy Centers Staff and the Energy Division Project

Manager for WE&T to select the two installer learning interventions that are most tied to impacting installation

practices and contributing to potential energy savings. In particular, we will leverage information on trainings

that we glean from our review of course materials (Task 1), installer surveys (Task 5) and the instructional

design assessment (Task 6) to develop questions that will establish a baseline of knowledge and installation

behaviors. We will build on these data in subsequent tasks to assess knowledge and behavior change that

may contribute to additional energy savings. This will include an in-person audit and instructor interviews

described below.

In-person Audit of Learning Interventions

In-person audits by the study team will allow us to assess whether:



◼ Learning intervention implementation is consistent with the design and intent of the course materials;

◼ Instructors are successfully using adult learning principles and practices in structuring and delivering

learning interventions; and

◼ How course materials support additional energy savings.

Conduct Instructor Interviews

Opinion Dynamics will conduct on-site instructor interviews as part of the in-person audit of learning

interventions, depending upon the instructor’s availability and willingness to participate in the interview. We

anticipate that on-site interviews may include discussions with 14 different instructors as there are some

courses that include multiple instructors.

These interviews will consist primarily of open-ended questions to allow for expansion on topics, and will focus

on:

◼ The instructor’s previous experience with and training in adult learning principles and practices;

◼ Ways in which instructors tailor the use of the materials to better meet the needs of the participants;

◼ Instructors’ assessments of how training may improve energy efficiency projects and lead to additional

energy savings;

◼ Special considerations that may affect findings from the in-person audit (e.g. whether this was a

“typical” delivery of the class or whether there were special circumstances); and,

◼ Instructors’ suggestions for refining the course design and materials and enhancing processes related

to design and delivery of courses within the program.

We will develop an interview guide to support these discussions to help ensure all relevant questions are

addressed during the conversation.

Deliverables: Evaluation yardsticks, draft and final interview guides, course-specific data from review of

materials, course-specific data from in-person audits.

Task 7: Assess Learning

In this task, we will assess Kirkpatrick’s Level 2 – Learning for the two selected installer focused HVAC learning

interventions. At this level, we will measure the degree to which participants change attitudes, increase

knowledge, and enhance skills as a result of the learning experience. For these two learning interventions, we

will develop pre- and post- tests administered via web survey to ascertain knowledge and skills gain and

application based on the learning objectives of the learning intervention. We will also explore the possibility of

using simulations (such as the HVAC Job Ready simulations that Interplay Learning is developing with the

California Community Colleges) to gain insight on ability to apply the knowledge and skills in real-world settings.

We will develop effective questions to address the relevant levels of Bloom’s Taxonomy7 that align with the

learning objectives. We will ask the Energy Centers to use these pre- and post-tests in all occurrences of the

two learning interventions in Year 2.

7 Bloom’s Taxonomy is a classification system used to define and distinguish different levels of human cognition including Knowledge,

Comprehension, Application, Analysis, Synthesis, and Evaluation.



Deliverables: Draft and final pre- and post-tests

Task 8: Assess Behavior

Using the same two installer focused learning interventions prioritized in Task 7, we will assess Kirkpatrick’s

Level 3 – Behavior. In this level, we measure the degree to which participants apply what they have learned

outside of the learning environment. While we will have the self-report data regarding application of learning

in the installer survey, we think it is essential to actually assess the application of the learning through direct

observation. To do this, we will recruit participants to conduct ride-alongs with to observe them in the field. We

will rely on our trained consulting staff to recruit participants for this effort, potentially using the utilities and

instructors to add credibility to our recruitment process. We will explain to participants that we are interested

in refining the curriculum for the class they participated in to reflect more real-world experience as opposed

to assessing their installation knowledge, skills, and abilities. We expect each session to last one day and will

work to schedule the day when the installer is working on installation activities (as opposed to invoicing or

other duties).

During each ride-along, Eric Shum or Tom Hines of Tierra Resource Consultants, will carefully observe the

installer using a rubric developed to align with the KSAs developed for Deliverable 30 and the learning

objectives of the learning intervention.8 During the down-time between appointments, Mr. Hines or Mr. Shum

will debrief with the participant about the last customer appointment to dive deeper into the intricacies of the

decision-making process as well as the installer’s motivations, and challenges. We will also use this

opportunity to ask about the installer’s definition of installation success and his or her experience with the

learning intervention. We have budgeted to conduct ride-alongs with up to 20 installers.

There are often differences between what people say and what they do in a real situation. This dual-pronged

method, which combines observation and interviewing, allows us an opportunity to identify these gaps. Mr.

Shum or Mr. Hines will take notes and obtain photographic and audio documentation when possible. We will

offer installers a $500 incentive in appreciation of their time.

Deliverables: Draft and final observation protocols

Task 9: Conduct Indirect Impact Evaluation

The California Protocols, as well as the International Performance Measurement and Verification Protocol

(IPMVP), outline methods for measuring indirect energy savings impacts. Applying these methods to education

and training efforts, however, poses several challenges as discussed earlier. One of these challenges is the

potential difference in training audiences. WE&T training programs often serve end-users and market actors.

End users are defined as individuals who are in a position to undertake direct behaviors that lead to energy

savings actions. For example, a facility manager is in a position to take energy savings actions in one or more

of the premises they manage. In contrast, market actors are defined as individuals that may influence energy

savings in multiple premises over a long period of time. Examples of such individuals include, architects,

designers, and contractors who are in a position to share the lessons learned through the WE&T experience

to others who did not participate in the intervention. For this study, we will focus on end-users of who are in a

position to take direct energy savings actions.

To conduct the indirect impact evaluation, we will use the data collected in earlier tasks to inform engineering

analysis to determine gross energy impacts. This is the process of asking participants detailed questions

8 A rubric is an evaluation tool which outlines performance criteria and definitions of multiple levels of achievement for each

performance criteria.



regarding actions taken after program participation, then using engineering analysis to estimate savings due

to these changes for each of the selected focus areas.

From our experience conducting impact evaluations on WE&T programs, we are well aware of the challenge

to collect sufficient information through a survey effort. One of the primary challenges to this approach is the

difficulty of collecting sufficient information regarding energy savings actions taken (e.g. replacement and

changes to O&M behaviors) within the short time frame of a telephone or Internet survey. However, we have

iterated survey content and question wording over time to maximize the reliability and validity of the data

collected through self-report surveys.

Since WE&T programs are often designed to channel attendees into other resource programs, we will also

measure cross-program attribution to ensure that gross savings from WE&T interventions are not also being

counted toward gross savings from other IOU EE programs or vice versa. We will assess cross-program

attribution of program savings through responses to the participant survey. Responses from this survey will be

used to identify whether any of the reported energy savings actions also utilized a rebate or incentive from an

IOU energy efficiency program. If so, the Opinion Dynamics Team will deduct the estimated energy savings

value from the total gross savings for the participant, using the equation below.

Gross Savings = Sum of Energy Savings Actions Estimates – Energy Savings Rebated through IOU EE program

The data gathered will inform gross impacts estimates, as findings will serve as inputs into engineering

algorithms. To support the engineering analysis, we will draw upon existing data sources such as the

Commercial Buildings Energy Consumption Survey (CBECS), the California Commercial End-Use Survey (CEUS),

and the Database of Energy Efficiency Resources (DEER) to determine savings for specific measures, as well

as to gather information on baseline load intensities among other uses.

Task 10. Reporting

The Opinion Dynamics Team will work with the Commission to develop draft and final reports in accordance

with CPUC specifications.

Deliverable: Draft and Final Reports

1.2.4 Study Timeline and Milestones

In Table 3, we identify the timeline for the study.

Table 3. Deliverable 26 Timeline

Task Q4

2018

Q1

2019

Q2

2019

Q3

2019

Q4

2019

Q1

2020

Q2

2020

Q3

2020

Q4

2020

Q1

2021

Q2

2021

Q3

2021

Task 1: Materials and

Database Review x x x x X x x x

Task 2: Energy Division

and WE&T Program

Staff Interviews

x x

Task 3: Assessing the

Use of NMEC

Methodology to Evaluate

WE&T Programs

x x x



Task Q4

2018

Q1

2019

Q2

2019

Q3

2019

Q4

2019

Q1

2020

Q2

2020

Q3

2020

Q4

2020

Q1

2021

Q2

2021

Q3

2021

Task 4: Review and

Refine Learning

Intervention Exit Survey

x x

Task 5: Conduct

Installer Surveys x x

Task 6: Instructional

Design Assessment x x x x

Task 7: Assess Learning x x

Task 8: Assess Behavior x x

Task 9: Conduct Basic

Indirect Impact

Evaluation

x x

Task 10: Reporting x x

In Table 4, we identify the milestones and total budget for the study.

Table 4. Deliverable 26 Milestones and Budget

Milestone Deliverable Budget Schedule

Milestone 1: Landscape

Analysis and NMEC (Tasks 1-3) NMEC Whitepaper

December

2019

Milestone 2: Course

Assessments (Tasks 4-6)

Course-specific data from in-

person audits July 2020

Milestone 3: Assessing

Learning and Behavior (Tasks

7 & 8)

Pre-tests, Post-tests, and

Observation Protocol

December

2020

Milestone 4: Impact Evaluation

and Reporting (Tasks 9 & 10) Final Report

October

2021

Total $525,000 October

2021

1.3 Deliverable 27 (WE&T-2): Partnerships with Training Institutions

Impact Evaluation

Program administrators are one of many organizations statewide that have interest in developing California’s

current and future energy workforce. The WE&T landscape is complex and diverse, spanning a number of high-

potential markets, delivery channels, occupations, and jobs. As the CLTEESP states, “An effective,

comprehensive WE&T program for a new energy efficient economy requires collaborative efforts by many

entities. It is not the core mission of utilities to effectuate the level of change needed to create a

comprehensive WE&T program, nor can ratepayers fully fund the effort.” The CLTEESP identifies government

agencies, educational institutions, community-based and non-profit organizations, and industry and labor

organizations as key partners in identifying and implementing effective workforce strategies to sustain and

grow a robust green energy economy.



In the business plans, program administrators continue to focus on partnerships with third-party entities as a

key cross-cutting strategy for meeting the state’s ambitious energy goals. However, interveners, such as the

Consortium of Energy Efficiency, posit that “Insufficient alignment between the WE&T programs and the

State’s core occupational training institutions, such as community colleges, apprenticeship programs, CSUs,

UCs, CBOs and workforce development boards, has been a major barrier to achieving the State WE&T goals.”

A review of WE&T annual reports also suggests that there is a wide variety of partnerships, but it is unclear

how these partnerships work together to achieve overall portfolio goals.

Thus, in order to understand the impacts of partnerships, we will need to define what a partnership is, and

what makes a partnership effective, before we can even begin to understand the impact outcomes such as

energy savings, job creation, or job placement.


The research objectives for the partnerships with training institutions impact evaluation are to:

◼ Define the term partnership;

◼ Identify partnership types;

◼ Map existing and planned partnerships to partnership types;

◼ Determine the evaluability of partnerships;

◼ Understand the functioning and impact of partnerships with WE&T training and job placement

organizations;

◼ Characterize how partnerships are being implemented;

◼ Determine and measure partnership effectiveness indicators (such as joint ownership and

accountability for results, and effective communication and collaboration strategies); and,

◼ Assess the impacts (such as job placement and gross energy savings) of two WE&T partnerships as

case studies.


Table 5 below identifies the methods we will utilize to meet the study objectives.


Study Objective

Data

Request and

Materials

Review

Program

Staff

Interviews

Define

Partnerships and

Catalog

Partnership

Types

Survey

Current

Partners

Job

Placement

Surveys and

Interviews

Conduct Impact

Evaluation

Define the term

partnership ✓ ✓ ✓

Identify partnership

types ✓ ✓ ✓ ✓



Study Objective

Data

Request and

Materials

Review

Program

Staff

Interviews

Define

Partnerships and

Catalog

Partnership

Types

Survey

Current

Partners

Job

Placement

Surveys and

Interviews

Conduct Impact

Evaluation

Map existing and

planned partnerships

to partnership types

✓ ✓ ✓ ✓

Determine

evaluability of

partnerships

✓ ✓ ✓ ✓

Understand the

functioning and

impact of

partnerships with

WE&T training and

job placement

organizations

✓ ✓ ✓ ✓ ✓

Characterize how

partnerships are

being implemented

✓ ✓ ✓

Determine and

measure partnership

effectiveness

indicators

✓ ✓ ✓ ✓ ✓

Assess the impacts

(such as gross energy

savings, job

placement and job

creation) of WE&T

partnerships

✓ ✓


Task 1. Initial Data Requests, Program Material Review, and Literature Review

Opinion Dynamics will complete a review of program materials such as business plans, implementation plans,

past WE&T evaluations, program theory and logic models (PTLM), and relevant policies such as SB-350. We

will also conduct a comprehensive literature review of how partnerships are defined, what constitutes a

partnership, what are the characteristics of a good partnership and how partnerships have been evaluated.

We will conduct a systematic search of the available literature on partnerships, including conference

proceedings, regional report databases, Google Scholar, and academic databases. We will use a set of key

words to search each database in a consistent manner. At the outset of this effort, we will define the criteria

for inclusion in the literature review, which will ensure that the literature review process is efficient, focused,

and relevant. Opinion Dynamics will assess the results of keyword searches with respect to relevance based

on abstracts or executive summaries. Papers or reports deemed applicable to the topic at hand will then be

selected for a full text review.

We will submit an initial request for relevant program materials, budgets and partnership data as available.

These documents may also include updated program performance metrics as well as associated goals,



participant tracking databases to date, and all available partnership plans, contracts, memorandums of

understanding, etc. Reviewing these materials will allow the team to:

◼ Understand current program design and implementation including current and planned partnerships

and how they fit into the program and market landscapes;

◼ Understand what data is currently tracked related to each of the WE&T sub-programs and how data

is shared among the PAs; and,


develop the questions that we want to pose to them.

We will leverage background information collection and review across all studies in this proposal to minimize

PA burden and maximize the efficient use of evaluation resources. Opinion Dynamics will conduct a thematic

analysis of each of the identified data sources above using NVIVO—a powerful qualitative analysis tool—to

uncover evidence-based insights and findings. Using NVIVO, we will mine the data fully and ensure that none

of the data collection information is lost as we progress through the study. As necessary, we will submit a

secondary data request for additional material identified as relevant to this evaluation through Task 2.

Deliverables: Data request

Task 2: Conduct Program Staff Interviews

We will conduct semi-structured interviews with PA WE&T staff to understand the role of partnerships in each

of the WE&T programs and subprograms. The objective of speaking with WE&T staff is to understand: (1)

partnership definitions, (2) roles and responsibilities related to partnerships, (3) current and planned

partnerships by program and sub-program, (4) partnership goals, (5) the partnership development process,

(6) partnership tracking data, (7) partnership best practices and lessons learns, (8) partnership theories of

change, and (9) partnership evolution given planned modifications to the portfolio. We will also use these

interviews to understand how programs are meeting regulatory guidelines of expanding/initiating partnerships

with entities that do job placement and requiring placement experience for any new partners in WE&T

programs and new solicitations.9 We will also explore the nuances of PA funding related to job placement.

We anticipate these interviews happening via telephone and lasting approximately 60-90 minutes each. Our

team plans to conduct 16 of these interviews total—8 in Year 1 and 8 in Year 2—to understand how things

have evolved and to inform our partnership program data tracking reports in both years.

Deliverables: Draft and final interview guides, second data request (if needed)

Task 3. Update PTLMs, Define Partnerships and Catalog Partnership Types

A key to ensuring that this study is successful is to answer the question: “What is a partnership?” For WE&T,

A17-01-013 Rev 1 indicates that a final metric for “Expanding WE&T Reach via Collaborations” is the “number

of partnerships by sector (complete partnership defined by curriculum developed jointly + agreement). While

that is one definition of a partnership, based on our prior work, we hypothesize that there are many types of

partnerships and corresponding outcome paths that would influence how to define an impact evaluation so

that it does not waste valuable time and resources on a premature or inappropriately designed evaluation.

From our experience, we hypothesize there are different types of partnerships based on funding structures,

9 A17-01-013 Rev 1 Proposed Decision



agreement types, contractual relationships, and if partners provide oversight to subcontractors or deliver

services directly to the end-user.

Since partnerships are a strategy as opposed to a program, we will also catalog how these partnerships are

implemented and theories of change related to this strategy. We hypothesize there are multiple outcome

pathways for partnerships such as the co-development of curriculum, train the trainer agreements, or the

lending of energy saving tools. Theories of change link outcomes and activities to explain the how and why

the desired outcomes will occur. In contrast, logic models describe program activities and link them to outputs

and outcomes. The business plans propose changes to the structure of their WE&T portfolios10 and thus we

will update our past program theory logic models (PTLMs) to reflect the current WE&T structure.

Using all of the data compiled in the past three tasks, we will develop: (1) a working definition of partnerships,

(2) a classification system to catalog partnerships into homogeneous categories, and (3) updated PTLMs that

reflect the proposed changed reflected in the business plans and current thinking. We will codify our findings

from the past three tasks in a memo. We will vet the definition, taxonomy, PTLMs and crosswalk with the

Commission and the WE&T Program Coordinating Group (PCG).

Deliverables: Draft and final memo, draft and final presentation for PCG meetings

Task 4: Conduct Evaluability Assessment

An evaluability assessment (EA) is essentially a pre-evaluation process that examines the extent to which a

program (or in this case program strategy) can be evaluated in a reliable and credible way. It is aimed at three

major purposes:

◼ Is there an adequate theory of change?

◼ Is there a logical theory of change?

◼ Are there measurable goals?

◼ Are the long-term impact and outcomes clearly identified?

◼ Does it provide plausible causal links toward goals?

◼ Are there clear terms and agreements that define the partnership relationship?

◼ Has the partnership theory been implemented as planned?

◼ Have the planned activities been completed in the manner required by the theory?

◼ Are there adequate data to test the causal connections and outcomes?

◼ Do the program records/tracking systems contain appropriate information including participants,

near-participants and their relevant contact information as well as data to inform proximal and distal

indicators?

◼ Is there data on a potential control group?

◼ Do data exist or is it possible to collect outcome data that can test goal attainment?

To determine the best areas of focus for examining impacts given a limited evaluation budget, we will conduct

an evaluability assessment of each of the partnership types identified in the taxonomy described in Task 3.

10 e.g. Southern California Edison proposes consolidating the three traditional WE&T subprograms--Connections, Planning, and

Centergies--into one program called Integrated Energy Education and Training (IEET); PG&E proposes the same consolidated program

(IEET) which will include Core Energy Education and Technical Upskill Interventions.



Through this evaluability assessment, we will determine which partnership types (defined in Task 3) and what

specific partnerships have the most potential in meeting long-term outcomes such as energy savings, job

creation, and job placement. This information will be codified in an interim memo; findings from which will

heavily influence subsequent research tasks.

Deliverables: Draft and final memo

Task 5: Survey Current Partners

In order to verify our partnership taxonomy and help to inform the development of our impact evaluation plan,

we will conduct an online survey of entities currently partnering with PA WE&T programs. In developing survey

instruments, we will rely heavily on information gleaned from previous tasks, in particular the evaluability

assessment (Task 4). We will attempt a census of all partners collecting the key partner representative and

his or her contact information through a data request. We will conduct this survey in Year 2. We estimate that

these surveys will take approximately 15 minutes. Since what we define as a partner will be determined in

Task 3, we are not sure as to the number of partners. For budgeting purposes, we are assuming 20 partners

per year.

The objectives of this survey are to understand, from the partner perspective, the functioning and impact of

partnerships, validate information received from the PA’s, understand from the partner perspective what data

is available, capture any concerns regarding sharing data, understand the purpose of a partnership,

understand the goals of the partnership, characterize perceived success of the partnership, identify concerns

with the partnership and uncover any lessons learned. We will utilize these data to verify the accuracy of the

data collected so far and inform the impact evaluation planning in Task 9.

Deliverables: Draft and final survey

Task 6: Develop Impact Evaluation Plan

Using the data from the evaluability assessment and the partner survey, Opinion Dynamics will work with the

Commission to finalize an evaluation plan for an indirect impact evaluation in accordance to the specifications

discussed in Task 9 of this work plan. We will focus on a plan to measure impacts that align with the outcomes

identified in the PTLMs developed in Task 3, along with indirect energy savings that may result in partnerships

with WE&T programs.

Deliverables: Draft and final work plans for basic indirect impact evaluation

Task 7: In-Depth Interviews with Employers

To assess job-related impacts, we will conduct semi-structured interviews with employers that work with

occupational training organizations currently partnering with PA WE&T programs. We will focus these in-depth

interviews on employers that work with organizations focused on job training and placement that we identify

in Task 5. The research team will conduct interviews by telephone with representatives of these employers to

measure program-related impacts, such as job placement success, employee performance, and retention.

Opinion Dynamics will coordinate with program and occupational training partner organizations to obtain a list

of employers where partners have successfully placed candidates from 2018 to the present.

The research team plans to conduct interviews with up to 12 employers engaged with partner organizations

that focus on job training and placement. Based on information gleaned through the partner survey (Task 5)

we will adjust the number of necessary interviews accordingly. To ensure sufficient response rates, we will



offer $100 incentives (in the form of an Amazon gift card, or a donation to pre-selected charitable

organizations on their behalf) to employers willing to complete interviews.


Task 8: Survey with End-Users Receiving Job Training

WE&T training programs often serve end-users and market actors. End users are defined as individuals who

are in a position to undertake direct behaviors that lead to energy savings actions or receive job-related

benefits as a result of WE&T program activities. For this task, we will focus on end-users of the partnership

who are in a position to experience the outcomes identified in the PTLM updates. Examples might include

increased rates of job placement, change their behavior to become more successful in their positions, and

take direct energy savings actions.

Opinion Dynamics will conduct surveys with these end-users that have received services from participating

partner organizations. We will work with partner organizations to obtain a list of the clients they served in

2018, 2019, and the first half of 2020 and will draw a representative sample of individuals across

participating institutions. The objectives of this survey are to further develop an understanding of the types of

services that partner organizations provide, assess how those services may be augmented through their

partnerships with WE&T programs, and gather data to measure job-related impacts from WE&T programs—

e.g., job placement, performance, retention, and other job-related metrics identified in the evaluability

assessment (Task 4). The research team will work with the Commission to select two types of partnerships

based on the results of the evaluability assessment (Task 4). We will work with the Commission to determine

appropriate selection criteria, such as size, length of relationships, and data availability to inform the selection

of the two partnerships for evaluation. We will strive for each partnership selected to represent a different

partnership type. The research team will then develop a sample frame based on the partnership types and the

populations they serve gleaned from the partner survey (Task 5).

Opinion Dynamics will administer the survey online and plans to complete up to 140 surveys with end-users.

However, upon receipt of tracking data from partner organizations, we will adjust the sample frame and

number of anticipated survey completes accordingly. To ensure sufficient response rates, we will offer $25

incentives (in the form of an Amazon gift card, or a donation to pre-selected charitable organizations on their

behalf) to employers willing to complete interviews.

Deliverables: Draft and final survey

Task 9: Conduct Impact Evaluation

To conduct an impact evaluation, we will use participant surveys and secondary data to inform analyses to

estimate job-related impacts, and other gross energy impacts. From our experience conducting impact

evaluations on WE&T programs, we are well aware of the challenge to collect sufficient information through a

survey effort. One of the primary challenges to this approach is the difficulty of collecting sufficient information

regarding energy savings actions taken (e.g. replacement and changes to O&M behaviors) within the short

time frame of a telephone or Internet survey. However, we have iterated survey content and question wording

over time to maximize the reliability and validity of the data collected through self-report surveys.

The estimation of job-related and energy impacts requires detailed information about the training participating

organizations received, the energy-saving actions that end-users took, and specifications about equipment

and the location where end-users took action. In addition, we will need to collect detailed information regarding



the change in job placement and retention rates of end-users, along with information on their performance

since participating in trainings supported by WE&T programs.

Since WE&T programs are often designed to channel attendees into other resource programs, we will also

measure cross-program attribution to ensure that gross savings from WE&T interventions are not also being

counted toward gross savings from other IOU EE programs or vice versa. We will assess cross-program

attribution of program savings through responses to the participant survey. Responses from this survey will be

used to identify whether any of the reported energy savings actions also utilized a rebate or incentive from an

IOU energy efficiency program. If so, the Opinion Dynamics Team will deduct the estimated energy savings

value from the total gross savings for the participant.

Gross BOC Savings = Sum of Energy Savings Actions Estimates – Energy Savings Rebated through IOU EE

program

To support this impact evaluation, we will rely heavily on previous tasks, specifically the evaluability

assessment, to outline theories of change and impact-related goals to measure. Further, we will gather data

from a variety of secondary sources to inform gross energy impacts estimates. This may include but are not

limited to the Commercial Buildings Energy Consumption Survey (CBECS), the California Commercial End-Use

Survey (CEUS), and the Database of Energy Efficiency Resources (DEER) to determine savings for specific

measures, as well as to gather information on baseline load intensities among other uses.

Deliverables: Estimates of job-related impacts and gross indirect energy impacts

Task 10. Reporting


to the specifications discussed in this proposal.



In Table 6, we identify the timelines and milestones for the study.


Task Q4

2018

Q1

2019

Q2

2019

Q3

2019

Q4

2019

Q1

2020

Q2

2020

Q3

2020

Q4

2020

Q1

2021

Q2

2021

Q3

2021

Initial Data Requests,

Program Staff Interviews x x x

Update PTLMs, Define

Partnerships and

Catalog Partnership

Types

x x x

Conduct Evaluability

Assessment x

Survey Current Partners x x

Develop Impact

Evaluation Plan x x



Task Q4

2018

Q1

2019

Q2

2019

Q3

2019

Q4

2019

Q1

2020

Q2

2020

Q3

2020

Q4

2020

Q1

2021

Q2

2021

Q3

2021

In-Depth Interviews with

Employers and End-Use

Surveys

x x

Conduct Impact

Evaluation x x

Reporting x x

Table 7 presents the milestones and total budget for this study.



Milestone 1: Landscape

Analysis, PTLMs, and

Partnerships Definition

(Tasks 1-3)

PCG Meeting January 2020

Milestone 2: Partner Survey &

Impact Evaluation Planning

(Tasks 4-6)

Survey and Impact Evaluation Plan September

2020

Milestone 3: Employer

Interviews and End-User

Survey (7 & 8)

Interview Guide and Survey March 2021

Milestone 4: Impact Evaluation

and Reporting (Tasks 9 & 10) Final Report

October

2021

Total $470,000 October

2021

1.4 Deliverable 28 (WE&T-3): WE&T Career Connections Process

Evaluation

Numerous studies and legislation have informed and shaped the strategy for WE&T as proposed in the

business plans, prompting program administrators to adjust their program offerings for better alignment and

coverages of California's energy efficiency industry, as well as redirect some of the program focus to

disadvantaged communities. For 2018, the PG&E WE&T program has been realigned to “better serve

California’s energy workforce” into three core strategies: K-12 Career Connections (CC), Career and Workforce

Readiness (CWR), and Integrated Energy Education & Training (IEET), with the other program administrators

having similar plans.11 IEET includes two sub-components: Core Energy Education and Technical Upskill. We

focus specifically on PG&E as the Career Connections program was designated a statewide program with PG&E

as the statewide administrator in D.16-08-019.

With the passage of SB350 in 2015, Section 8 of this bill ordered the Commission to consider green workforce

training specifically calling out "workforce development and job training for residents in disadvantaged

neighborhoods, including veterans, at-risk youth, and members of the state and local community conservation

11 Pacific Gas & Electric. “Energy Efficiency Business Plan 2018-2025.” Jan 2017. Accessed May 3, 2018.

https://docs.wixstatic.com/ugd/0c9650_cbeb1d9e14cf4575845e8d5cd6bce57f.pdf



corps” and coordination with the “California Workforce Investment Board, the Employment Training Panel, the

California Community Colleges, and other entities to ensure a qualified, well-trained workforce is available to

implement the program requirements.”12 The California’s Workforce Education and Training Needs

Assessment (Needs Assessment) report completed in 2011 outlined two goals regarding workforce training

issues:13

◼ Energy savings, which requires that participating contractors and workers have the skills they need to

ensure that equipment is properly installed, commissioned, and maintained, and that buildings are

designed, constructed, and retrofitted consistent with best practice and technical specifications for

energy efficiency; and,

◼ Inclusion of workers from disadvantaged communities in rewarding careers in energy efficiency.

To achieve these goals, the Needs Assessment offered three recommendations for the IOUs:14

◼ Incorporate a set of contractor and workforce standards and other interventions into the program

requirements for energy efficiency incentive programs.

◼ Redesign WE&T programs to better align with and leverage California's main training and education

institutions, in order to incorporate EE-specific skills and knowledge in the broader skills set of

workers in key occupations.

◼ Create an inclusion program to broaden opportunities for workers from disadvantaged communities

to enter rewarding careers related to energy efficiency.

This new structure aims to achieve the goals outlined in SB350 and the Needs Assessment by ensuring that

current and potential members of California's energy efficiency workforce have quality education and training

at every stage of development, while also specifically targeting members of disadvantaged communities to

give them opportunities for rewarding careers in the energy efficiency industry.

1.4.1 Program Background

According to the most recent implementation plan, filed in 2013 and renewed for the 2016/17 program, the

Connections program has historically consisted of seven program components, five of which were geared

toward teachers and students of different age groups within the K-12 sector and two of which were geared

towards the college sector. In line with the planned WE&T program restructuring in 2018, the updated Career

Connections program will cater specifically to teachers and students in the K-12 sector.

The purpose of this program is to support teachers and organizations training future generations of the energy

workforce by providing teaching materials and resources to schools and teachers to educate students about

energy and sustainability fundamentals, and by providing green career awareness and exploration resources.

As such, the Career Connections program has the following overarching objectives15:

12 California State Legislature. Clean energy and pollution reduction act of 2015 (SB-350). October 2015. Accessed May 3, 2018.

https://leginfo.legislature.ca.gov/faces/billHistoryClient.xhtml?bill_id=201520160SB350 13 Donald Vial Center on Employment in the Green Economy - Institute for Research on Labor and Employment - University of

California, Berkeley. “Workforce Issues and Energy Efficiency Programs: A Plan for California’s Utilities.” May 2014. Accessed May 3,

2018. http://laborcenter.berkeley.edu/pdf/2014/WET-Plan-Executive-Summary14.pdf 14 Donald Vial Center on Employment in the Green Economy - Institute for Research on Labor and Employment - University of

California, Berkeley. “California Workforce Education and Training Needs Assessment.” 2011. Accessed May 11, 2018.

http://laborcenter.berkeley.edu/pdf/2011/WET_Part1.pdf 15 Pacific Gas & Electric. 2018-2025 PG&E Energy Efficiency Business Plan. Jan. 2017. Accessed May 3, 2018.




◼ Educate K-12 students on energy and sustainability fundamentals;

◼ Increase awareness of energy and sustainability career pathways for high school students and

career-seeking adults; and,

◼ Support Title 1 K-12 schools as a way of supporting disadvantaged communities.

Study Objectives

The CC program was designated as a statewide program with PG&E as the statewide administrator. Current

plans indicate that an implementer for this program will not be selected until early 2021. Thus, this study will

focus on an evaluability assessment for this program. The research objectives of the CC process evaluation

are to:

◼ Develop/Refine a CC logic model; and,

◼ Assess the evaluability and adaptability of the CC Program.


The table below identifies the methods we will utilize to meet the study objectives.


Research

Objectives

Secondary Data

Review

Program

Staff

Interviews

Program Theory

and Logic

Model

Evaluability

Assessment

Develop/Refine

a CC logic

model

✓ ✓ ✓ ✓

Assess the

evaluability

and

adaptability of

the CC Program

✓ ✓ ✓ ✓


We provide detailed descriptions of each task below.

Task 1. Data Requests and Secondary Data Review

Opinion Dynamics will complete a review of program administrator business plans and program materials

including Program Implementation Plans (PIPs) and past process evaluations, key overarching studies such

as the Needs Assessment, and relevant policies such as SB-350. We will submit an initial request to the

program administrator (PG&E) for relevant program materials, budgets, and participant data as available.

These documents may also include updated program performance metrics as well as associated goals,

participant tracking databases to date, and all available curriculum plans.

Reviewing these materials will allow the team to:



◼ Develop a base understanding of the current program design and implementation including any sub-

components and how they fit into the marketplace;

◼ Identify and develop a base understanding of current program theory and logic models; and,


develop the questions that we want to pose to them.

As necessary, we will submit a secondary data request for additional material identified as relevant to this

evaluation through Task 2.

Deliverables: Initial data request

Task 2. Program Staff Interviews

We will complete semi-structured in-depth interviews via telephone with key stakeholders from the

Commission, the program administrator (PG&E), program implementers, and any other key design and

implementation staff identified in Task 1. We will use these interviews to build our understanding of the

planned design and implementation structure of the CC program and performance metrics beyond what can

be gleaned from available program documentation. Specifically, we will focus on the following topics:

◼ Understanding the planned program design and implementation process, changes planned, updates

to proposed program metrics, associated goals and budgets;

◼ Understand alignment between the CC program and other WE&T programs;

◼ Alignment between the program and Strategic Plan goals;

◼ Characterize workforce outcomes; and,

◼ Ensure that we identify all relevant documentation related to the Career Connections programs for

review.


Task 3: Program Theory and Logic Model

Opinion Dynamics will develop/refine a PTLM that accurately documents the program design, delivery, and

theory of change for this iteration of the CC program. A PTLM captures the program’s intended outcomes and

links those outcomes to specific metrics and activities, mapping the causal relationships between the different

program components. As such, our team feels that this is a critical step in an initial evaluation; one that will

create a foundation for program planning, service delivery, and future evaluations.

We understand that PTLMs are complicated and representing the Program’s components accurately and

thoroughly may take several different iterations. Our team will draw upon information gleaned from interviews

with program staff and the review of secondary data (as outlined in Task 1) to complete the initial logic model.

Our team will then present the draft logic model to the program team and revise the PTLM as necessary to

ensure that the final logic model accurately captures the CC Program’s design and theory of change. We will

continue to work with the program team until all agree that the PTLM is sufficiently detailed, and will therefore

be useful, before finalizing.

Deliverables: Draft and final program theory and logic models



Task 4: Evaluability Assessment

As part of the evaluation of the CC Program, Opinion Dynamics will complete an evaluability assessment to

help the Commission and the program team ensure that the CC Program addresses the needs of California’s

disadvantaged workers and make progress toward the Program’s intended outcomes.

Our evaluability assessment will cover three main topics: (1) assess if there is an adequate program theory

that provides plausible causal links toward program goals; (2) consider if the program theory has been planned

appropriately (e.g., if the program is properly staffed and being implemented as intended); and (3) determine

if the planned program tracking database will collect adequate data to support the causal connections and

outcomes. To complete this EA, we will draw upon information gleaned from previous research tasks to

determine if established metrics align with program goals, and if those metrics are measurable. During

interviews with program staff, our team will also assess how data are currently collected and recommend ways

that the program team may be able to augment data collection in the future. Additionally, through this task,

our team will recommend that the program capture any additional data necessary to adequately evaluate the

CC Program in the future. The goal of this exercise is to assist the Commission with building an evidence-base

to support future evaluations and to ensure that the CC Program is able to show progress toward program

goals.

Task 5. Reporting

Opinion Dynamics will work with the Commission to develop draft and final reports in accordance with CPUC

specifications.

Deliverable: Draft and final reports


The table below summarizes the evaluation timeline and deliverables for the Career Connections evaluation.


Task 2020 2021

Q4 Q1 Q2 Q3 Q4

Data Requests and Secondary Data Review x

Program Staff Interviews x

Program Theory and Logic Model x x

Evaluability assessment x x

Reporting x x

Table 10 presents the budget by milestone for this study.





Milestone 1: Evaluability

Assessment Report Evaluability Assessment Report October 2021

Total $50,000 October 2021

1.5 Deliverable 29 (WE&T-4): Career and Workforce Readiness

Process Evaluation

The Career Workforce Readiness (CWR) Program is new to the WE&T portfolio of programs. The program will

be implemented primarily through partnerships with different community-based organizations (CBOs),

including building-related job training programs and workforce development agencies. The Career and

Workforce Readiness program was designated as a statewide program with PG&E as the statewide

administrator. Current plans indicate that an implementer for this program will not be selected until early

2021. Thus, this study will focus on an evaluability assessment for this program.

The new CWR sub-program was created in response to SB350, passed in 2015. SB350 states that, in addition

to establishing a regulatory framework and a comprehensive set of programs to achieve greater energy savings

for California residents, the Commission shall promote greater penetration of programs within disadvantaged

communities, including programs aimed at workforce development. In addition, SB350 specifically notes that

the Commission should work with existing workforce development organizations that serve disadvantaged

communities.16 Working in concert with the existing WE&T and workforce development programs, the central

purpose of the CWR program is to support members of the energy workforce through training-related

collaborations with workforce development organizations whose primary mission includes serving

disadvantaged communities. The CWR program is a statewide WE&T program.

The CWR Program aims to integrate existing workforce development organizations’ resources—case

management, soft skills training, job placement, etc.—with technical energy education and resources. Its

central objectives are as follows:17

◼ Support the integration of energy efficiency content into existing workforce development training

programs whose primary audience is disadvantaged communities and workers.

◼ Support trainers of programs supporting disadvantaged workers and communities in support of

overarching energy efficiency portfolio objectives.

◼ Provide energy and green career awareness resources to workforce development organizations

serving disadvantaged workers and communities.

◼ Provide energy and resource conservation education materials and resources to organizations who

train disadvantaged workers.

The CWR Program is intended to facilitate entry from disadvantaged communities into the energy efficiency

career pathway. As such, its funding cannot be used to fund energy efficiency education and training activities

that are already occurring, or to fund activities that workforce development agencies and organizations are

16 California State Legislature. Clean energy and pollution reduction act of 2015 (SB-350). October 2015. Accessed May 3, 2018.

https://leginfo.legislature.ca.gov/faces/billHistoryClient.xhtml?bill_id=201520160SB350 17 Pacific Gas & Electric. 2018-2025 PG&E Energy Efficiency Business Plan. January 17, 2017. Accessed May 3, 2018.




funded to do (i.e., case management, job placement, etc.). It will fund the addition and integration of energy

efficiency funding to some existing efforts.

As the impetus for the CWR Program is to support disadvantaged workers, coming to some agreement on a

definition for this core component of the program’s target audience was of particular importance. According

to California’s Health and Safety Code, the California State Legislature broadly defines “disadvantaged

communities” as those that live within areas that meet the following criteria:18

◼ Areas disproportionately affected by environmental pollution and other hazards that can lead to

negative public health effects, exposure, or environmental degradation.

◼ Areas with concentrations of people that are of low income, high unemployment, low levels of

homeownership, high rent burden, sensitive populations, or low levels of educational attainment.

For the purposes of developing programs to support SB350, however, the IOUs built on this definition to

specifically describe the CWR Program’s target audience.

In late 2015, with input from stakeholders, the IOUs arrived at a definition of a disadvantaged worker as an

individual who meets at least one of the following three criteria: 1) lives in a high unemployment zip code

where unemployment rate is at least 150% of the median unemployment rate for the county or for the state;

or 2) lives in a low income zip code where the average household income is 50% below Area Median Income

(AMI); or 3) has a referral from a collaborating community-based organization (CBO), state agency, or workforce

investment board.19


The research objectives of the CWR process evaluation are to:

◼ Develop a CWR logic model; and,

◼ Assess the evaluability of the CWR Program.


Table 11 below provides a high-level overview of Opinion Dynamics’ suggested research approach and how

each activity aligns with the study objectives outlined above.


Research Objectives Secondary

Data Review

Program Staff

Interviews

Program

Theory and

Logic Model

Evaluability

Assessment

Develop a CWR logic model

✓ ✓ ✓ ✓

Assess the Evaluability ✓ ✓ ✓ ✓


18 California Health and Safety Code. Chapter 4.1. Section 39711 (a) 19 Pacific Gas and Electric Company. “Energy Efficiency Business Plan 2018-2025”. 2017. Accessed May 6, 2018.




In the remainder of this section, we describe each of our suggested research tasks in detail. For each, we

discuss our approach, how we envision each task addressing the research objectives outlined previously, and

any deliverables associated with that task.

Task 1. Review of Program Materials, Tracking Data, and Secondary Data

To begin, Opinion Dynamics will complete a review of secondary materials, including any existing program

materials, policy documents, business plans, program tracking data, and any other related research studies.

We will also gather any relevant secondary data that are publicly available.

Through this task, the research team will develop a base understanding of the current program design,

program goals, intended outcomes, and identify key program staff that we will interview during Task 2.

Additionally, we will use this task to gather additional policy context that inform the desired outcomes.

Materials reviewed as part of this task will include, but are not limited to:

◼ Program Implementation Plan and any other program manuals

◼ Final Business Plan

◼ Senate Bill 350

◼ Survey of Inclusion Workforce Landscape

◼ The California Long-term Energy Efficiency Strategic Plan

◼ Other regulatory materials or studies related to WE&T and disadvantaged communities

Deliverables: Initial data request

Task 2: In-Depth Interviews with Program Staff

We will complete semi-structured in-depth interviews with key program design and program management staff

identified in Task 1 via telephone. We will use these interviews to build our understanding of the theories of

change, the planned design and implementation structure of the CWR Program, and performance metrics

beyond what can be gleaned from available program documentation. We will also use this interview as the

primary data source in developing the CWR program theory and logic model (PTLM). Specifically, we will focus

on:

◼ Understanding the planned program design and implementation process, along with the initial

program goals;

◼ Accurately capturing the CWR Program’s intended outputs, outcomes and theory of change;

◼ Gathering information about synergies between the CWR Program and other WE&T programs;

◼ Alignment between the program and Strategic Plan goals, and best practices outlined in the Survey

of Inclusion Workforce Landscape; and

◼ Ensuring that we identify all relevant documentation related to the CWR Program for review.

Deliverables: Draft and Final Interview guides



Task 3: Program Theory and Logic Model

As this is the first process evaluation of the CWR Program, Opinion Dynamics will develop a PTLM that

accurately documents the program design, delivery, and theory of change. A PTLM captures the program’s

intended outcomes and links those outcomes to specific metrics and activities, mapping the causal

relationships between the different program components. As such, our team feels that this is a critical step in

an initial evaluation; one that will create a foundation for program planning, service delivery, and future

evaluations.

We understand that PTLMs are complicated and representing the Program’s components accurately and

thoroughly may take several different iterations. Our team will draw upon information gleaned from interviews

with program staff and the review of secondary data (as outlined in Task 1) to complete the initial logic model.

Our team will then present the draft logic model to the program team and revise the PTLM as necessary to

ensure that the final logic model accurately captures the CWR Program’s design and theory of change. We will

continue to work with the program team until all agree that the PTLM is sufficiently detailed, and will therefore

be useful, before finalizing.

Deliverables: Draft and final program theory and logic models

Task 4: Evaluability Assessment

As part of the first evaluation of the CWR Program, Opinion Dynamics will complete an evaluability assessment

(EA) to help the Commission and the program team ensure that the CWR Program addresses the needs of

California’s disadvantaged workers and make progress toward the Program’s intended outcomes.

Our evaluability assessment will cover three main topics: (1) assess if there is an adequate program theory

that provides plausible causal links toward program goals; (2) consider if the program theory has been planned

appropriately (e.g., if the program is properly staffed and being implemented as intended); and (3) determine

if is the planned program tracking database will collect adequate data to support the causal connections and

outcomes. To complete this EA, we will draw upon information gleaned from previous research tasks to

determine if established metrics align with program goals, and if those metrics are measurable. During

interviews with program staff, our team will also assess how data are currently collected and recommend ways

that the program team may be able to augment data collection in the future. Additionally, through this task

our team will recommend that the program capture any additional data necessary to adequately evaluate the

CWR Program in the future. The goal of this exercise is to assist the Commission with building an evidence-

base to support future evaluations and to ensure that the CWR Program is able to show progress toward

program goals.

Task 5. Reporting


to the specifications discussed in this proposal.



The table below summarizes the evaluation timeline and deliverables for the Career and Workforce Readiness

process evaluation.




Task 2020 2021

Q4 Q1 Q2 Q3 Q4

Review of Program Materials, Tracking Data, and

Secondary Data x

Interviews with Program Staff x

Program Theory and Logic Model x x

Evaluability Assessment x x

Reporting x x




Milestone 1: Evaluability

Assessment Report Evaluability Assessment Report October 2021

Total $50,000 October 2021

1.6 Deliverable 30 (WE&T-5): Knowledge, Skills, and Abilities Market

Studies

The CALTEESP sets forth a goal of establishing "energy efficiency education and training at all levels of

California's educational system." As codified in their business plans, program administrators continue to drive

toward the CALTEESP vision for utility WE&T programs--"By 2020, California's workforce is trained and fully

engaged to provide the human capital necessary to achieve California's economic energy efficiency and

demand-side management potential.”20 WE&T is a key cross-cutting strategy for providing the knowledge to

recognize EE opportunities and the skills and abilities to take action on those opportunities. California has

passed ambitious energy efficiency and greenhouse gas legislation with SB350 requiring California to double

energy efficiency savings and reduce greenhouse gas levels 40% below 1990 levels by 2030. Similar

statewide mandates include Assembly Bill 758 and Assembly Bill 802, which both address energy efficiency

performance and reporting for buildings. These statewide mandates create a demand for new and more

precise energy efficiency and whole building skillsets in the California workforce.

Numerous studies have identified a critical gap between ex ante and ex post energy savings estimates. Many

of these studies identify poor quality work as a key contributor to this gap, especially in the areas of HVAC

installation and maintenance, advanced lighting control systems, and weatherization services. In the

Workforce Issues and Energy Efficiency Programs: A Plan for California Utilities report, the Don Vial Center

states that quality of work is an important, under recognized factor for "meeting short-term savings targets

and long-term energy efficiency goals." The report goes on to say that "persistent quality problems in Heating,

20 California Public Utilities Commission. “California Long Term Energy Efficiency Strategic Plan” Sept. 2008, updated Jan. 2011.

Accessed May 8, 2017. http://www.leginfo.ca.gov/pub/15-16/bill/sen/sb_0301-0350/sb_350_bill_20150911_enrolled.pdf



Ventilation, and Air Conditioning (and Refrigeration) (HVAC) and other key building systems point to a need for

greater efforts to ensure quality. Despite the common use of upfront workforce standards in other industries

like health care, there is a notable lack of substantive qualifications required of contractors and workers who

participate in most IOU programs. Reliance on upfront standards for participating contractors and workers can

be particularly effective at promoting work quality because they attract high-performing contractors and screen

out (or improve the quality of work of) lower performing contractors."21

In the recent Responsible Contractor Policy for EE Programs: Market Intelligence Study,22 nearly all of the 78

stakeholders Opinion Dynamics interviewed discussed the topic of training and the need for consistent job

definitions, skill requirements, and training standards to support energy and demand reduction goals.

Concerns centered on defining how jobs are defined and what key knowledge, skills, and abilities (KSAs) are

needed to perform a particular task. Who defines these KSAs? How do we ensure KSAs are kept current with

quickly changing technology? How do you keep knowledge and content up-to-date when faced with contractors

who are providing on-the-job training and instructors who are teaching courses for equipment that did not exist

when they were certified? How do you ensure that training includes skills that are relevant to energy efficiency

when energy efficiency is not necessarily a focus of credentialing organizations?

Reviewing program theory and logic models across resource programs finds that an implicit assumption of

most programs is that contractors and technicians--with experience--have the right KSAs to install measures

in a way that will realize and ideally maximize energy savings. Programs and WE&T efforts utilize training to

support this assumption; however, the key challenge is defining what the right KSAs are. In programs where

these training needs are incorporated into the PTLM, program staff often consult relevant industries for

guidance. This is a challenge as many industries do not have standard career paths and competency models,

nor does their training adequately incorporate energy efficiency components.

Let's briefly look at a specific example in the HVAC arena. The CALTEESP specifies the following HVAC goal:

"Quality HVAC installation and maintenance become the norm. The marketplace understands and values the

performance benefits of quality installation and maintenance." Unfortunately, the CALTEESP does not define

the terms "quality installation” or “quality maintenance." Stakeholders involved in developing the CALTEESP,

indicate that “quality” was defined in development discussions as adherence to industry standards, such as

the ACCA/ANSI Quality Installation Specification 5 (ACCA 5) and ASHRAE/ACCA/ANSI Standard 180.

Unfortunately, that detail was not codified in the CALTEESP. The HVAC Action Plan, which was designed to help

the HVAC sector achieve the CALTEESP goals, identifies training as a critical success factor, stating that

"California must set standards for quality work." This document mentions industry standards and the need to

align them with Title 24, but still does not define the terms “quality installation” and “quality maintenance”.

Given that many program administrators took part in the development processes for both the CALTEESP and

the HVAC Action Plan, many took relevant industry standards as the definition of quality installation and quality

maintenance and used them to design HVAC programs. For example, the HVAC Optimization program was

based on ASHRAE's Standard 180 Standard Practice for Inspection and Maintenance of Commercial Building

HVAC Systems. The Residential Quality Installation program was based on ACCA 5. These standards provided

an important framework for program design and implementation. However, standards traditionally define what

tasks to perform and sometimes what order to perform these tasks in, but they do not define how to perform

the tasks nor what tools to use to perform the task. Given that quality installation and maintenance is

21 Don Vial Center, “Workforce Issues and Energy Efficiency Programs: A Plan for California’s Utilities.” May 2014. Accessed May 8,

2017. http://laborcenter.berkeley.edu/pdf/2014/WET-Plan14.pdf

22 Opinion Dynamics, “Responsible Contractor Policy for EE Programs: Market Intelligence Study.” Accessed December 31, 2017.

http://www.calmac.org/publications/Responsible%5FContractor%5FPolicy%5FStudy%5FReport%5FFINAL%2Epdf



considered a highly technical activity in which improper execution of the necessary steps can lead to incorrect

diagnoses of problems as well as solutions that can decrease efficiency, these industry standards do not

necessarily lead to energy savings in isolation. The Western HVAC Performance Alliance (WHPA) has

recognized this gap and has worked to develop “User Guides” to operationalize the “how” of these standards.

Given that this is a significant undertaking, however, these passionate volunteers have only made moderate

progress.

In HVAC, this lack of clarity on what knowledge, skills, and abilities are needed to perform standards-based

work has led to challenges with conducting evaluations. One of the challenges with past HVAC impact

evaluations is a discrepancy between the evaluator’s perspective on how to conduct key maintenance tasks

and the procedures the program was teaching its contractors. This lack of consistency has recently led to

controversy regarding ex post savings estimates and evaluation recommendations, such as with Work Order

32.

The lack of clarity on workforce standards and identified KSAs both at the industry level, as well as at the

whole building design level will likely continue to undermine the effectiveness of investments in programs and

training. With all of this in mind, Deliverable 30 is crucial to advancing California’s ambitious energy goals, and

to putting a stake into the ground that all stakeholders can work toward.


Jobs can be best understood as a series of tasks and responsibilities that an employee conduct. A task is an

action designed to contribute to a specified end result. KSAs are what enable employees to perform these

tasks. Often, these terms are used interchangeably; yet they represent distinctly different dimensions of a job.

Here we define these terms in context of this study:

◼ Knowledge is the body of information needed to perform a task. It focuses on the actual understanding

of particular concepts. It is theoretical and not practical. A person can read a state’s drivers manual

on how to drive a car, but have no practical experience driving a car.

◼ Skills reflect the practical application of the theoretical knowledge. They reflect capabilities or

proficiencies developed through training or actual experience. Once a person passes their Department

of Motor Vehicle’s written test, they obtain a learner’s permit to develop the proficiency of driving a car

prior to taking the driving test.

◼ Abilities are underlying, enduring traits useful for performing tasks. They are often related to personal

and social attributes that tend to be innate. There is a fine line between skills and abilities. Driving a

car would require the ability to safely perform a range of physical activity to manipulate the controls

for starting, stopping, and backing up.

The objective of this research is to identify the KSAs needed for implementers, contractors, technicians, and

contractor laborers on energy efficiency and whole building design concepts within specific industries.


We will conduct a Job Task Analysis to meet the objectives of this study. Job Task Analysis is the systematic

investigation of work activities, worker attributes, and work context. This study is particularly focused on

understanding what tasks (work activities) and what knowledge, skills, and abilities (worker attributes) are



essential to energy efficiency for implementers, contractors, technicians, and contractor laborers in specific,

high-impact industries. Work context, defined as situational opportunities and constraints that affect the

occurrence of behavior (i.e. time allowed to conduct maintenance on an HVAC system is a work context

attribute that affects what maintenance tasks are completed and with what thoroughness), is not a focus of

this research.

Significant research on job analysis has taken place in the past 75 years. There are many methods for

conducting job analyses. Some of the most common include observations, individual interviews, facilitated

meetings, and questionnaires. Time requirements, cost, and intended use of information must be considered

when choosing job analysis methods.

Given the timeframe for this study, and the goal of identifying work activities and worker attributes, Opinion

Dynamics has selected the DACUM (Developing a Curriculum)23 Job Task Analysis method as the foundation

for this research. We have selected this method as it is a time-tested approach that is effective, comparatively

inexpensive and relatively fast. This method also lends itself to conducting a job analysis to a specific portion

of a total job—in this case those portions involving energy efficiency and whole building principles. DACUM is

based on three premises:24

1. Expert workers can describe and define their job more accurately than anyone else. While some methods

suggest using supervisors, managers, or content experts, this method purports that the people working

full-time in their positions are the real experts on the job.

2. An effective way to define a job is to precisely describe the tasks the expert workers perform.

Understanding what expert workers do to be successful at is key to understanding a job and how to train

others for that job.

3. All tasks, in order to be performed correctly, demand the use of certain knowledge, skills, and abilities.

Knowledge, skills, and abilities are the enablers that make it possible for workers to be successful.

For each identified key area of study and job role (e.g. HVAC contractors, electricians), we will follow the

following process outlined in Figure 3. We have budgeted to conduct two job role analyses.

Figure 3. DACUM Job Task Analysis Process

23 The term DACUM is a misnomer. While initially developed as a method to conduct needs assessment to inform curriculum

development, it has been used for over 40 years to conduct job analyses for other purposes such as defining job descriptions,

developing HR strategies, making compensation decisions, creating selection assessments, meeting ISO 9000 requirements, and

developing career paths. It is used in schools, colleges, government agencies, military organizations, business and industry. 24 Wojton, M., Heimlich, J. “Professional Learning Framework: DACUM Panels.” 2016. Accessed May 12, 2018.

http://www.islframework.org/wp-content/uploads/2017/07/For-Posting-DACUM-Panel-Feedback-Report.pdf




Task 1: Identify Key Areas of Study

Opinion Dynamics will coordinate with the Energy Division Project Manager for WE&T to select two high impact

areas, for which we will identify necessary knowledge, skills, and abilities (KSA) for contractors and

technicians/contract laborers. We will also include implementers who act as contractors/technicians as

participants in the study process. Opinion Dynamics will base its recommendations on past evaluation

guidance, policy direction, program administrator business plans, labor market data, and expert opinion.

Potential areas of study include KSAs on HVAC installation and maintenance, heat pumps, home area

networks (HAN) or smart technology, with a key focus on how each of these are incorporated into efficient

whole building design. Given the differing KSAs required for residential vs commercial equipment we will also

work with the Energy Division Project Manager to focus the scope of each study area on one sector or the

other.

Task 2: Conduct Pre-Inventory Data Analysis

We will conduct a pre-inventory data analysis to gain an understanding of existing information and data

sources relevant to the selected areas of study. This analysis will help ensure that we:

◼ Use appropriate and familiar language in the development of workshop stimuli and homework;

◼ Prioritize key functions within a job that focus on energy efficiency;

◼ Align this study with existing industry-specific job frameworks to increase industry buy-in; and,

◼ Maximize the return on research investment.

We will also use this secondary analysis to understand the complexity of the occupations under study to further

refine the measures, applications, and careers to be examined within the project scope and to inform our

sample design. For example, there are many workers employed in direct HVAC positions, while others are

scattered among thousands of workers in Sheet Metal, Pipes, and Stationary Engineering trades. While the

KSAs are likely similar across these trades, we will want to include a representative from these trades in our

expert panels to ensure we capture any key nuances.

Drawing on our extensive experience, we will utilize known sources of data as well as conduct a brief literature

review to ensure our analysis includes all key information. We will assess each source to ensure the

information is up to date. We have also budgeted for eight in-depth interviews to collect additional data and

context should the need arise.

Known Sources

We will review job information from labor market sources such as the Occupational Information Network

(O*NET). The O*NET database—sponsored by the U.S. Department of Labor--contains hundreds of

standardized and occupation-specific descriptors on almost 1,000 occupations covering the entire U.S.

economy. It encompasses the broadest scope of work information ranging from labor market data and wages

to knowledge, skills and required tasks. The O*NET content model is the foundation of the system that

identifies the most important types of information about work. Figure 4 displays O*NET’s content model, with

the four elements we will emphasize in this study highlighted in blue.



Figure 4. Content model for O*NET

For example, O*NET includes the job--Heating and Air Conditioning Mechanics and Installers—and identifies

the information for each of the elements in the above content model. It also identifies 37 alternate occupation

titles.

We will also consult the Department of Energy and the National Renewable Energy Laboratory’s Guidelines for

Home Energy Professionals Project. This project conducted job task analyses, defined standard work

specifications that address minimum acceptable outcomes, developed accredited training programs, and

created Home Energy Professional certifications. These guidelines are currently utilized by the DOE’s

Weatherization program (WAP) and potentially include roles that could be considered in this study. Through

Opinion Dynamics’ in-depth assessment of this work for the Responsible Contractor Policy Market Intelligence

study, we are able to leverage the information as well as the lessons learned from their process. Such lessons

include the fact that workers need both requisite specialized task level skill expectations and broad training

that allows workers to take a holistic view of the systems they work with, enabling them to make situational,

systems-based decisions. We will consult the Bureau of Labor Statistics and the California Labor Market

Information Division for additional labor market and economic data as well as Opinion Dynamics’ PY2013-

2014 California Statewide Workforce Education and Training Program Contractor Training Market

Characterization

In addition to national and state-based labor market resources, we will consult specific work on the identified

areas of study. For example, for HVAC, we will review key standards such as ACCA Standard 5 and

ACCA/ASHRAE Standard 180. We will also consult the Knowledge Areas of Technician Expertise (KATEs)

identified by North American Technician Excellence (NATE), as well as the competency and task lists developed

by HVAC Excellence. We will utilize the compendium of work developed by the Western HVAC Performance

Alliance with a specific focus on the whole building career lattice work and the WE&T Gaps Reports.



Additional Sources

Opinion Dynamics will conduct a brief literature review to understand any additional research that can expand

on what we already know. We will conduct a systematic search of the available literature regarding the areas

of study, including conference proceedings, regional report databases, Google Scholar, and academic

databases. We will use a set of key words to search each database in a consistent manner. At the outset of

this effort, we will define the criteria for inclusion in the literature review, which will ensure that the literature

review process is efficient, focused, and relevant. Opinion Dynamics will assess the results of keyword

searches with respect to relevance based on abstracts or executive summaries. Papers or reports deemed

applicable to the topic at hand will then be selected for a full text review.

Opinion Dynamics will conduct a thematic analysis of each of the identified data sources above using NVIVO—

a powerful qualitative analysis tool—to uncover evidence-based insights and findings. Using NVIVO, we will

mine the data fully and use the information to develop the pre-work for the participants in each DACUM

workshop.

Deliverables: Draft and final Pre-Work

Task 3: Workshop Sample Strategy and Recruitment

Recruiting the right participants is the foundation for an effective DACUM job analysis. For each job focus

within each area of study, we will identify ten to twelve participants, with the goal of the final group consisting

of seven to ten participants so that we have a sufficient size to function effectively. Since we will be asking

them for a thirty-minute phone call, completion of a homework assignment and two days of their time, we will

offer participants a $1000 stipend as a thank you for participating in the study. The DACUM workshop cannot

succeed without the help of expert workers because they can describe their daily performance in terms of

activities and tasks they perform better than anyone else. Since we will be focusing on the energy efficiency

and building performance elements of each job, we will likely recruit past program participants in each key

area of study. The following selection criteria will guide our sampling strategy:

1. Technical Competence: Individuals selected should be highly skilled and aware of current

developments and needs in their industry.

2. Energy Efficiency/Whole Building Experience: Individuals need to be experienced with energy

efficiency and whole building principles as this study will focus on those aspects of the job.

3. Effective Communicator: Individuals selected must be able to explain what they do in a precise

and accurate way. Since the workshop involves brainstorming and consensus seeking, individuals

must be able to actively listen and participate in small group discussions.

4. Commitment: Individuals must commit to the process and actively participate.

5. Freedom from Bias: Individuals should be open-minded and free of bias.

We aim to balance screener length with researcher objectives to ensure respondent burden is minimized—

placing major qualifiers and disqualifiers (except for more sensitive demographic questions) at the beginning

of the screener. Prior to commencing the recruiting effort, the contractor will develop a draft recruiting guide

for all identified targets, which we will finalize based on comments from the Energy Division Project Manager

for WE&T.

Deliverables: Draft and final sampling strategy memo, draft and final screener



Task 4: Design and Facilitate DACUM Workshop

Once we have recruited our participants, the facilitators will reach out to participants via telephone to establish

rapport and orient them to the DACUM process, talk them through what to expect, discuss our expectations of

them, review logistics, discuss the pre-work, and answer any questions. We anticipate this discussion to take

approximately 30 minutes. We have found that this initial phone call is a key step to ensuring the workshop

goes well.

Following this phone call, we will send the participants the pre-work materials. We have found that in order to

get the most out of the DACUM workshop, providing a pre-work exercise is a best practice. This will be a short

exercise asking them to review the output of the task and KSAs from the pre-inventory analysis and

determining if the task belongs to that job, is an energy efficiency or whole building opportunity, and if the

KSAs are appropriately defined. Through this pre-work, participants are given the opportunity to reflect on the

job under study, which often adds depth and dimension to the discussions during the in-person workshop. All

pre-work exercises will be finalized based on comments from the Energy Division Project Manager for WE&T.

We will facilitate two DACUM workshops and will utilize either a Commission or Energy Center meeting room

to save rental costs. The DACUM workshop is a two-day in-person exercise in which we identify general areas

of job responsibilities, identify specific tasks performed related to that job responsibility, and develop a list of

KSAs necessary to conduct the specific tasks. For example, a general area of job responsibility might be

“maintain automobile” and the tasks that make up that responsibility would include “change motor oil,”

“change windshield wipers,” and “fill gas tank.” We will design key questions and activities to solicit this

information focusing on:

◼ Problems that a worker must solve

◼ Products or services that a worker produces

◼ Decisions that a worker makes

◼ Equipment that a worker operates

◼ Information that a worker collects, analyzes, or prepares

The facilitator's role is to set the stage, develop an energetic and comfortable environment, orient the

committee to the process, to guide them through the analysis, to draw out ideas, to question each proposed

task statement, to gain consensus from the group on each item, and to keep the discussion on target and the

process on schedule.

During the session, the high-performers brainstorm and discuss their work, coming to consensus about areas

of responsibility and required tasks. The facilitator actively assists the panel members by providing guidance

in the formulation of tasks statements, ensuring equal participation by all members, and coaching them to

describe their work in measurable and observable terms. We will utilize a storyboarding process, where

participants are encouraged to develop activity and task statements that include a verb, modifier, and a noun.

The storyboard cards are temporarily attached to the wall or board to allow for rapid changes, sequencing, and

modification. The storyboard process is very fluid, flexible and visual—all of which helps maintain the group's

focus on describing their job duties and tasks (see Figure 5). Important to the moderation is ensuring that task

statements meet key criteria such as having the right verbs, objects, and appropriate qualifiers. For example,

while the task "mow the lawn" is quite clear, the task statements "develop a plan" and "assess performance"

are not. What type of plan is needed, who are we to assess, and in what area? A committee often has to be

asked repeatedly for one or more words which help make the task statement "crystal clear."



Figure 5. DACUM Storyboarding Process

Source: Saskatchewan Institute of Applied Science and Technology

The facilitators are aware of the common pitfalls associated with DACUM workshops, which are (1) the

unsuitable composition of “expert worker” committees, (2) lack of definitions or criteria for job tasks and task

statements, (3) heavy dependence on outdated literature reviews, (4) lack of high-quality tasks statements,

and (5) the all-too-common failure to separate out tasks, knowledge, skills, and abilities. The facilitators will

utilize effective orientation materials, handouts, examples and moderation techniques to ensure these

hazards are avoided.

We will end the workshop with an evaluation of the workshop proceedings so that we can continuously improve

on the workshop design and facilitation iteratively.

Deliverables: Draft and final pre-work exercises; Draft and final DACUM process and facilitation guide; Draft

and final DACUM workshop evaluation; Draft activities, tasks and KSAs

Task 5: Verification

Following the workshop, we will solicit feedback through an online survey to validate the activities, tasks, and

KSAs identified through the workshops. The objective of the verification process is to ensure that we covered

all key job responsibilities, associated the right tasks with these responsibilities, and identified the correct

knowledge, skills and abilities need to perform these tasks, and to ensure that the results are appropriate for

a wide audience. We will advertise these surveys using online forums as well as target installers for which we

have email addresses.

Deliverables: Verified activities, tasks, and KSAs

Task 6: Reporting

Opinion Dynamics will work with the Commission to develop draft and final reports in accordance with CPUC

specifications.

Deliverable: Draft and final reports


In Table 14 we identify the timeline for the study.



Table 14. WE&T 30 Study Timeline

Q4

2018

Q1

2019

Q2

2019

Q3

2019

Q4

2019

Q1

2020

Identify key areas of study x x

Pre-Inventory Data Analysis X X

DACUM Workshop X X

Verification x x

Reporting x x




M1: DACUM Workshop

Design

Final DACUM process and facilitation

guide May 2019

M2: Final Report Report January 2020

Total $345,000 January 2020

California Public Utilities Commission Energy Efficiency Program Evaluation … Group B... ·...

Documents

Transcript of California Public Utilities Commission Energy Efficiency Program Evaluation … Group B... ·...