Duffy ECB Working Paper
Transcript of Duffy ECB Working Paper
-
8/3/2019 Duffy ECB Working Paper
1/68
Evaluation Capacity 1
Running head: EVALUATION CAPACITY BUILDING
Theory and Research on Evaluation Capacity Building
Jennifer L. Duffy
University of South Carolina
-
8/3/2019 Duffy ECB Working Paper
2/68
Evaluation Capacity 2
Theory and Research on Evaluation Capacity Building
The use of participatory, collaborative and empowerment approaches to
evaluation is a growing area of interest in within the field, in part in order to increase the
utilization of evaluations conducted (Preskill, Zuckerman, & Matthews, 2003).
Empowerment evaluation in particular emphasizes the importance of building capacity
for evaluation, and suggests that the community should own the evaluation (including
making decisions about evaluation questions and design) and should also learn how to
conduct their own evaluations (Wandersman et al., 2005). However, while interest in this
area is growing (as is suggested by a growing number of recent publications and the
recently started topical interest area in the American Evaluation Association), evaluation
capacity building (ECB) has only recently been distinguished as an area of work distinct
from the work of carrying out evaluation (Baizerman, Compton, & Stockdill, 2002a).
The purpose of this paper is to synthesize research and theory relevant to ECB,
not a particular approach to evaluation or evaluation theory. It is important to keep in
mind that while certain schools of evaluation (particularly collaborative, participatory,
and empowerment evaluation) explicitly make ECB a part of their work to varying
degrees, ECB is not limited to a particular brand of evaluation. ECB can be applied to
more traditional forms of evaluation as well, particularly ECB strategies that focus on
increasing demand for and utilization of evaluation (as has been done in international
ECB work). However, while ECB is not necessarily tied to collaborative, participatory, or
empowerment evaluation approaches, the majority of the research identified has
approached ECB from these perspectives.
-
8/3/2019 Duffy ECB Working Paper
3/68
Evaluation Capacity 3
In the following sections, definitions and theoretical issues relevant to ECB are
described, methodological challenges to research on ECB are identified, and research on
ECB is reviewed. In the final section, theoretical issues related to ECB will be
reexamined in light of the research reviewed, and further questions for research on ECB
will be identified.
Definitions of Key Concepts
This paper begins with a review of definitions of evaluation capacity and
evaluation capacity building. The concepts being examined within this paper are
complex, abstract, and cut across multiple disciplines, making the specification of the
definitions used particularly important.
Defining Evaluation Capacity
Several definitions of evaluation capacity are offered in the literature. Boyle,
Lemaire, and Rist (1999) define evaluation capacity as the human capital (skills,
knowledge, experience, etc.) and financial/material resources necessary for doing
evaluation (p. 5). Milstein & Cotton (2000) define evaluation capacity as the ability to
conduct an effective evaluation (i.e. one that meets accepted standards of the discipline)
(p. 1). They also propose a framework that includes five main elements of evaluation
capacity: motivational forces, organizational environment, workforce or professional
development, resources and support, and learning from experience. Gibbs, Napp, Jolly,
Westover, & Uhl (2002) define evaluation capacity as the extent to which a CBO has the
necessary resources and motivation to conduct, analyze, and use evaluations (p. 261).
The General Accountability Office (2003) defines evaluation capacity within federal
agencies as the ability to systematically collect, analyze, and use data on program
-
8/3/2019 Duffy ECB Working Paper
4/68
Evaluation Capacity 4
results and identifies key components of that ability as an evaluation culture, access to
high quality data and evaluation expertise, and collaborative partnerships with state and
local agencies who carry out the agencies work.
These definitions vary in the level of analysis to which they are applied. These
levels of analysis have implications for how ECB is defined and understood. Boyle and
colleagues (1999) definition is not anchored to a specific level; it can be applied to
individual-level evaluation capacity, or the capacity of a team, organization, or broader
system (like a regional or national government). Elements of the Milstein & Cotton
(2000) definition also can be applied across different levels, but the focus on elements of
the organizational environment suggest the necessity of going beyond the individual
level. Gibbs et al. (2002) explicitly place evaluation capacity at the organizational-level.
The basic definition provided by GAO is applicable across multiple levels, but some of
the specific elements of evaluation capacity identified are specific to the work done by
federal agencies. Each of these definitions includes resources, and two definitions also
explicitly include motivation as an aspect of evaluation capacity.
Arnold (2006) suggests that in understanding the evaluation capacity of the
individuals within an organization is necessary for understanding the evaluation capacity
of the organization. She suggests that individual evaluation capacity may be viewed on a
continuum ranging from doubters (who are skeptical of the value of evaluation) to
evaluation scholars (who are experts on evaluation). According to this framework,
organizational capacity for evaluation is reflected by an organization which has many
evaluation practitioners, with few doubter or scholars.
Defining Evaluation Capacity Building
-
8/3/2019 Duffy ECB Working Paper
5/68
Evaluation Capacity 5
Definitions ofevaluation capacity building found in the literature tend to be
complex, multifaceted, and laden with jargon. In addition, some literature (particularly in
the international development field) refers to evaluation capacity developmentand in
some cases use it interchangeably with ECB. Schaumberg-Muller (1996) used a broad
definition of ECB in a survey of international development donor agencies, including:
activities which provide support for systems of evaluation, audit, feedback and learning
from policies, programmes or projects performed at various levels it excludes activities
aimed solely at planning and appraisal activities (p. 5). The most common ECB
activities identified on that survey included training, technical assistance, and the
provision of equipment.
Boyle et al. (1999) do not define ECB, but do define evaluation capacity
development as activities and initiatives taken to implement an evaluation regime and
define evaluation regime as the configuration of evaluation capacity, evaluation practice,
organizational arrangements, and institutionalization within a national or regional
government (p. 6). In other words, Boyle et al. define evaluation capacity development as
activities intended to promote evaluation capacity and practice within governments,
including attending to where evaluation is placed within the government and
whether/how evaluation is institutionalized.
The editors of a special issue of an evaluation journal dedicated to the explication
of ECB developed a complex conceptual definition of ECB based upon their knowledge
and the available literature in the area (Baizerman, Compton, & Stockdill, 2002a;
Stockdill, Baizerman, & Compton, 2002). This conceptual definition is quite complex
and contains 11 distinct elements. Stockdill et al. define ECB as a context-dependent,
-
8/3/2019 Duffy ECB Working Paper
6/68
Evaluation Capacity 6
intentional action system of guided processes and practices for bringing about and
sustaining a state of affairs in which quality program evaluation and its appropriate uses
are ordinary and ongoing practices within and/or between or more
organizations/programs/sites (p. 8). While Stockdill et al. state that each of these 11
elements is important in helping to distinguish ECB from evaluation practice, they also
provide a simplified working definition of ECB: ECB is the intentional work to
continuously create and sustain overall organizational processes that make quality
evaluation and its uses routine (p. 14).
These definitions of ECB (and evaluation capacity development) share several
element. Each identifies ECB as an activity separate from actually carrying out
evaluations. In addition, Stockdill et al. (2002) and Boyle et al. (1999) both note the
importance of institutionalizing and evaluation practice within organizations. There are
also some key distinctions between the definitions. The conceptual definition provided by
Stockdill et al. focuses on the importance of sustaining the use of evaluation over time,
and emphasizes the importance of considering the context where the ECB is to take
place. Boyles definition is focused on building evaluation capacity within governmental
contexts, and as such it focuses on the importance of examining the legal authorization
for evaluation and which branch of government has responsibility for evaluation
activities. In contrast, the definition provided by Stockdill and colleagues focuses on the
organizational level, though it is also applicable to ECB that covers multiple
organizations. None of these definitions explicitly mentions building evaluation capacity
at the individual level, though some ECB activities mentioned by Schaumberg-Muller
(1996), such as training, are typically applied at the individual level.
-
8/3/2019 Duffy ECB Working Paper
7/68
Evaluation Capacity 7
Key Concepts Relevant to Evaluation Capacity Building
Evaluative inquiry and organizational learning.
The concept of evaluative inquiry deliberately ties the work of ECB to the idea of
organizational learning. Preskill and Torres (1999) define evaluative inquiry as an
ongoing process for investigating and understanding critical organizational issues that
is fully integrated with an organizations work practices, and as such, it engenders (a)
organization members interest and ability in exploring critical issues using evaluation
logic, (b) organization members involvement in evaluative processes, and (c) the
personal and professional growth of individuals within the organization. Torres and
Preskill (2001) suggest that this integration of evaluation logic and processes into the
regular work of the organization increases the utility of evaluation by explicitly moving
beyond evaluating the effects of specific programs or initiatives and instead focusing on
changing the functioning of the organization itself (which affects all activities of the
organization). They distinguish evaluative inquiry from similar fields of evaluation (like
empowerment evaluation and participatory evaluation) based on the focus on integrating
evaluative activities into the existing work process, ongoing learning at multiple levels,
cultural shift, and valuing of different perspectives.
Evaluative inquiry might then be conceptualized as a specific form of ECB at the
individual, organizational, and team levels. Rather than carrying out an evaluation, the
evaluator functions as a change agent or coach, facilitating the process of inquiry and
setting up processes to support this work. The social constructivist theory of learning
guides the specific mechanisms identified for promoting individual, team, and
organizational learning.
-
8/3/2019 Duffy ECB Working Paper
8/68
Evaluation Capacity 8
Preskill and Torres (1999) emphasize that organizational infrastructure that
supports learning is necessary for evaluative inquiry to be incorporated into the work of
the organization. In particular, they identify having an organizational culture that is
supportive of asking questions, taking risks, and learning from mistakes as a key part of
this infrastructure.
Process use of evaluation.
Process use of evaluation is one concept closely related to ECB. Process use
refers to the idea that in addition to the use of evaluation findings (e.g. using information
about outcomes to make decisions about program or policy effectiveness), the process of
participating in an evaluation may have additional effects for individuals and
organizations (Patton, 1997). At the individual level, process use can result in changes in
thinking and behavior (e.g. a shift to thinking in a more evaluative way about program
activities). Organizational level changes in procedure or culture may also result from
process use (e.g. institutionalizing the regular collection of program data and using it for
program improvement). Patton identifies four main ways evaluation logic and processes
can be used: enhancing stakeholders shared understanding of the activity being
evaluated, supporting the program, increasing participants engagement and ownership,
and program or organizational development. Building upon the definition of process use
outlined by Patton and their work assessing the impact participating in evaluation
activities had on staff members of an organization they worked with, Forss, Rebien, &
Carlsson (2002) identified five types of process use: learning to learn, developing
networks, extending communication, strengthening the project, and boosting morale.
-
8/3/2019 Duffy ECB Working Paper
9/68
Evaluation Capacity 9
They also point out that these different types of process use are likely to affect different
stakeholders in the evaluation in different ways.
Patton (1997) emphasizes that changes due to process use may be an unintended
ripple effect that is incidental to the evaluation process, or they may be intentionally
incorporated into the evaluation process. A review of empirical examples of process use
suggested that this type of use often occurs due to luck (Forss et al., 2002). However,
process use can be planned for and it has been theorized that such preparation is likely to
increase the amount of process use that takes place (Forss et al. 2002; Patton, 1997;
Preskill & Torres, 2000). It has been noted that the degree of involvement in evaluation
by organization members likely affects the amount of process use that takes place, since
those who have a high degree of involvement and ownership have the most exposure to
the evaluation process and thus are more likely to be affected (Patton, 1997). Preskill and
Torres suggest that in order to promote greater learning from participation in the
evaluation process, evaluators should take a clinical approach to evaluation, work to
span the evaluation-program boundary to become more involved in the development of
program activities, and assess the readiness for organizational learning from evaluation of
the groups with whom they work.
Milstein and Cotton (2000) explicitly include process use/learning from
experience participating in evaluation as an element of their evaluation capacity
framework. Stockdill et al. (2002) suggest that in order for an activity to be classified as
ECB, it must be intentional work aimed at building capacity (not accidental or incidental
learning). This does not conflict with the idea of process use, since that can be
-
8/3/2019 Duffy ECB Working Paper
10/68
Evaluation Capacity 10
intentional or unintended, and Patton recommends more explicitly building opportunities
for process use into evaluation activities.
Cousins, Goh, Clark, and Lee (2004) present a theoretical framework that
integrates evaluative inquiry within an organizational learning system. They suggest that
both evaluative inquiry and use of evaluation (particularly process use) contribute to the
development of evaluation capacity within the organization, and that over time, through
sustained evaluative inquiry and continued and routine use of evaluation finding and
processes, evaluation capacity becomes integrated within organizational culture (p. 107).
ECB thus is seen as a pathway to a broader organizational change or shift in
organizational culture. This model suggests that evaluation activities precede the
development of evaluation capacity, which precedes a broader shift in organizational
culture (or the development of an evaluation culture). This contrasts with theorists who
suggest that evaluation culture is a necessary precursor of the development of evaluation
capacity.
Mainstreaming evaluation.
Another concept closely tied to ECB is mainstreaming evaluation, or making
evaluation a part of organizations everyday activities, including organizational culture,
work ethic, and job responsibilities (Sanders, 2003). Duignan (2003) suggests that in
order for evaluation to be mainstreamed, people at all levels throughout our
organizations and policymaking processes are being more evaluative about what they
do they must have appropriate evaluation skills, systems, structures, and resources to
support them in taking a more evaluative approach to their work (p. 12). Duignan
emphasizes that it is not necessary for people to call what they are doing evaluation in
-
8/3/2019 Duffy ECB Working Paper
11/68
Evaluation Capacity 11
order for evaluation to be mainstreamed. This approach requires evaluators to give
evaluation away (i.e. to share the skills and tools of evaluation with people who are not
professional evaluators). This necessitates building the evaluation capacity of the
individuals and organizations that are expected to mainstream evaluation into their work.
Grudens-Schuck (2003) suggests that the mainstreaming of evaluation amounts to
the incorporation of evaluation into the organizational culture. She points out that there
are benefits and problems associated with evaluation becoming a part of organizational
culture. Once something has become a part of organizational culture, it is difficult to
change, and it may take considerable effort to get people to unlearn what they already
know so they are able to take on and use new knowledge (e.g. to begin using a new
approach to evaluation or to think about evaluation in a new way).
Key Theoretical Issues Relevant to Evaluation Capacity Building
Relationship between evaluation practice and evaluation capacity building.
Stockdill and colleagues (2002) in particular emphasize the importance of making
a distinction between the practice of evaluation itself (whether or not it involves the
participation of other stakeholders) and activities where the intent is to build evaluation
capacity. While it has been suggested that the act of participating in an evaluation can
lead to important changes not directly related to the evaluation findings (see e.g. process
use of evaluation, discussed above), Stockdill et al. make the case that ECB goes beyond
simple evaluation practice. They emphasize that the practice of ECB is not (or should not
be) a matter of luck or incidental learning that takes place within the context of an
evaluation. Simply carrying out an evaluation (or a series of evaluations) is not enough to
-
8/3/2019 Duffy ECB Working Paper
12/68
Evaluation Capacity 12
build the capacity of an organization to incorporate and sustain evaluation as a regular
part of daily activities.
Demand for evaluation and the creation of an evaluation culture.
A culture supportive of evaluation activities is frequently identified as a necessary
element (or important precursor) of ECB (Boyle, 1999; Boyle et al., 1999; Preskill &
Torres, 1999; Owen, 2003). Boyle and colleagues have proposed a demand and supply
framework for evaluation capacity development. This framework suggests that in order
for evaluation to be institutionalized, both ability to carry out evaluation (or supply of
evaluation) and interest in utilizing evaluation (or demand for evaluation) are necessary.
Where the demand for evaluation is weak, evaluation is unlikely to be used, even when
the financial and human resources necessary to carry out evaluation are in place. Boyle
(1999) suggests that developing educated consumers of evaluation will increase demand
for evaluation services by professional evaluators and can also lead to an evaluation
ethos where evaluation is valued as an integral part of the governmental decision-
making process (p. 145). He recommends training and other support be provided to
potential evaluation users to increase their ability to be involved in evaluation, including
determining how evaluation can be useful and how to use the evaluation results. The
framework outlined by Boyle et al. suggests that the demand for and supply of evaluation
should be developed more or less simultaneously, because when either one outpaces the
other, the evaluation results yielded are undesirable (because either high quality
evaluation is not available or the evaluation resources are unlikely to be used.)
Owen (2003) proposes a tentative theoretical model of the creation and causation
of evaluation culture, based upon his experience working with organizations which have
-
8/3/2019 Duffy ECB Working Paper
13/68
Evaluation Capacity 13
developed such a culture and research on dissemination of implementations. He suggests
that introducing evaluation to an organization is similar to the process of introducing any
other major innovation (e.g. by ensuring that the benefits of evaluation are clear to staff
members). According to this model, action by the organizational leadership and specific
aspects of the way evaluation is implemented internally contribute to the development of
an evaluation culture in which there is a role for evaluation in decision-making within an
organization (p. 43). Similar to the model proposed by Cousins et al. (2004), Owens
model suggests that implementation of evaluation activities within an organization (at
least in a limited way) precedes the development of a culture which supports evaluation.
Owen also notes that resistance to efforts to change an organizations culture is one
possible barrier to the development of an evaluation culture.
According to Toulemonde (1999), an evaluation culture provides the collective
pressure that makes decision-makers overcome their reluctance, even when evaluation
deeply contradicts their self-interest. He suggests that evaluation culture can be built
through intense, sustained communication of the importance of evaluation by well-
respected messengers, or what he calls preaching or a sermon in favor of evaluation.
The message is that evaluation is an important component of sound public
management. Suggested methods of communication include workshops, newsletters,
conferences and journals.
Trochim (2005) does not offer specific suggestions for how to develop an
evaluation culture, but he identifies a number of elements of what an evaluation culture
might look like (from the perspective of evaluators). In particular, he suggests that and
evaluation culture should be teaching-oriented; emphasizing the connections between
-
8/3/2019 Duffy ECB Working Paper
14/68
Evaluation Capacity 14
evaluation and everyday thinking, and encouraging all staff members to become involved
in evaluating what they and their organizations do. He notes that this requires the
development of evaluations that are relatively simple, practical, and easily understood
and carried out by non-professional evaluators. It also requires resources to educate staff
members about evaluation.
Stages of Evaluation Capacity.
Several authors have suggested that it is useful to identify levels or developmental
stages of evaluation capacity. Love (1991) identifies six developmental stages of internal
(organizational) evaluation capability: ad hoc evaluation, systematic internal evaluation,
goal evaluation, effectiveness evaluation, efficiency evaluation, and strategic benefit.
Loves stages refer not to an organizations ability to carry out evaluations, but instead to
its ability to plan evaluations and use information collected from them in a way that
maximizes their benefit to the organization. At the first stage of evaluation capability (ad
hoc evaluation), the value of evaluation is recognized by managers, but is used in an
isolated way to evaluate specific projects, and there is typically little systematic
collection of data that could be useful for evaluation. In subsequent stages, evaluation
data is collected in increasingly more systematic ways and with greater attention to its
utility, evaluations seeks to answer more challenging questions, and evaluation results are
tied more closely to organizational planning. At the sixth level (strategic benefit, the
highest identified by Love) evaluation is used by senior managers to examine the
strategic value of activities to the company and to the broader society.
A different model of stages of organization-level evaluation capacity was
developed by Gibbs et al. (2002). They developed this model based on information
-
8/3/2019 Duffy ECB Working Paper
15/68
Evaluation Capacity 15
gathered through qualitative interviews about evaluation with community-based
organizations and health department staff and technical assistance providers who work
with them. Three stages of evaluation capacity were identified: compliance, investment,
and advancement. At the compliance stage, funding agency requirements are the primary
motivator of evaluation activities. Community based organizations may lack the
resources necessary for more than basic compliance with requirements, or they may see
little value in evaluation activities (or even view them as a burden that takes away from
program activities). At the investment stage, the primary motivation for evaluation
activities is program improvement. Organizational leadership supports the use of
evaluation, and organizational resources are devoted to making sure that staff members
have the time and skills necessary to carry out evaluation activities. At the advancement
stage, there is broad organizational support for evaluation, and more sophisticated
evaluations focused on demonstrating the outcomes of their programs are carried out.
Significant resources are devoted to evaluation, and these organizations may partner with
an external evaluator or have staff members who have specialized evaluation skills.
Professionalization of Evaluation
There are multiple viewpoints regarding whether evaluation should be primarily
in the hands of professional evaluators, or whether evaluation should be presented as an
activity that can and should be carried out as part of daily organizational practice. In
general, authors who approach ECB from the international development perspective
suggest that the development of a class of professional evaluators (including formal
education for evaluators) is a necessary component of evaluation capacity (Boyle, 1999;
Toulemonde, 1995). Toulemonde even offers a system for classifying evaluators from
-
8/3/2019 Duffy ECB Working Paper
16/68
Evaluation Capacity 16
amateurs (who perform some evaluation, but are limited by a lack of knowledge/ability to
apply different evaluation techniques) to professional evaluators (who have specialized
training in multiple methods and frameworks for evaluation and spend the majority of
their work time on evaluation). In contrast, Duignan (2003) cautions that the
professionalization of evaluation can reify evaluation and make it a separate activity that
may or not be done to programs rather than a making evaluation a part of everyday
organization activities (p. 12) and thus counter to the goals of ECB as defined by
Stockdill et al. (2002) and mainstreaming of evaluation as outlined by Sanders (2003).
These differences regarding how professionalization of evaluation relates to ECB
make sense when considering the different perspectives from which ECB has been
approached. From an international development perspective, a major concern is the
predominance of evaluators from outside the host country (primarily meeting the
evaluation needs of donor agencies, not the country where the programs being evaluated
are taking place). Given that context, the focus on developing professional evaluators
who can conduct evaluation of aid programs funded by donors and help generate demand
for evaluation within their own country makes a great deal of sense. In settings where
there is already a supply of in-county evaluators, there is a shift in focus to giving
evaluation away by building the capacity needed for organizations to make evaluation a
part of their ongoing daily work.
Situations where evaluation capacity building may be contraindicated.
Some authors have questioned whether ECB is an appropriate strategy when
working with organizations with limited resources. Miller, Kobayashi, and Noble (2006)
suggest that particularly for small community-based organizations, building the capacity
-
8/3/2019 Duffy ECB Working Paper
17/68
Evaluation Capacity 17
to carry out their own evaluation may be unsustainable. Particularly for smaller
organizations with few resources, learning about and carrying out evaluation may be
viewed as a burden on staff. They suggest instead a model they call insourcing
whereby organization staff members are trained in a few basic evaluation-related
activities (e.g. logic model development, data collection and entry) to minimize the
burden placed on staff members. Evaluation activities which require more specialized
knowledge (like the development of the evaluation design and data analysis) would then
be carried out by professional researchers or evaluators. Miller et al. suggest that in
insourcing, the cost of hiring external evaluators to do this work can be decreased by
creating networks of community based organizations that have similar types of programs
and need similar types of evaluation instruments. The evaluator can create design
standard evaluation tools across these organizations, spreading the cost across the groups
participating. Insourcing is in a sense a hybrid of more traditional evaluation by external
evaluators and models which require more intensive capacity building from within an
organization.
Miller et al. (2006) suggest that minimizing the burden of evaluation on
organization staff members and focusing their participation around the activities most
likely to be useful and interesting to them (creating the program logic model and
interpreting the evaluation results) is more likely to develop support for evaluation among
staff members than is more intense involvement that is viewed as a burden that takes
away from other duties (like service provision).
Methodological Challenges
-
8/3/2019 Duffy ECB Working Paper
18/68
Evaluation Capacity 18
ECB has been characterized as a field in its infancy(Stockdill et al., 2002) and
as such the field is struggling with issues of definition and measurement of constructs like
evaluation capacity, as well as defining what activities are considered ECB. This is
further complicated by the extraordinarily multi-national and multidisciplinary nature of
the field. Studies reviewed here span ECB efforts on 4 different continents and in fields
as diverse as education, international development, HIV prevention, and substance abuse
prevention.
A related challenge is that the development of measures of evaluation capacity (at
the individual and organizational levels) is in early stages. A review of the existing
measures of evaluation capacity is beyond the scope of this paper, but it is worth noting
that a number of potential measure of evaluation capacity at the individual and
organizational levels have been put forward in recent years. A particular challenge to
developing measures of evaluation capacity is the degree to which ECB work is context
specific (Stockdill et al., 2002). Measures developed for ECB using a particular
framework or for use in a particular setting may need to be adapted in order to be used by
other researchers.
Wing (2004) identifies a number of challenges inherent in assessing the
effectiveness of capacity building initiatives, including the abstract nature of measureing
capacity, the dynamic and constantly changing nature of capacity building, lack of clarity
about whose goals should be measured and whether the outcome of interest is knowledge
or behavior change, and the tension between the need to document results and the
gradual, long term nature of building capacity.
Research on Evaluation Capacity Building
-
8/3/2019 Duffy ECB Working Paper
19/68
Evaluation Capacity 19
As was noted in the introduction, ECB has relatively recently been identified as a
separate field of inquiry and research. A number of authors have identified a need for
further empirical research on ECB (Baizerman et al., 2002b; Cousins et al., 2004). Much
of the existing research to date consists of reflective case studies which are based
primarily on the reflections of the authors on the experience of ECB (Cousins et al.).
These accounts are most often from the evaluators perspective, and provide varying
levels of detail about what the capacity building provided entailed, whose capacity was
build, and what evidence there is to support their assertions. Some provide considerable
explanation of the context where capacity building took place, the approach taken by the
evaluator, and how results of ECB are being evaluated (e.g. Compton et al., 2002). Other
case study authors mention that ECB took place in the context of a collaborative or
participatory evaluation, but provide little detail about what was done to help
participants build capacity for carrying out evaluation (e.g. Biott & Cook, 2000).
In addition to the growing number of reflective case studies on ECB, some
researchers have begun to use quantitative and qualitative methods to further
understanding of the effects of ECB efforts. As noted in the previous section, there are a
number of methodological challenges to examining capacity building efforts, particularly
related to assessment of evaluation capacity. The studies reported here are largely
preliminary efforts. Relatively few of the studies reviewed here look at the effects of
capacity building across multiple levels (e.g. at the individual and organizational levels),
though several do (Campbell et al., 2004; Chinman et al., 2006; Stevenson et al., 2002).
Most have small sample sizes, and several authors do not clearly report on the number of
individuals included in analyses (Brown & Reed, 2002; Nagao et al., 2005). Only one of
-
8/3/2019 Duffy ECB Working Paper
20/68
Evaluation Capacity 20
the studies examined here included a comparison group (Chinman et al., 1996); the others
relied primarily on single group pre-post comparisons or qualitative methods. An
additional concern is that while definitions of ECB focus largely on the integration of
evaluation into the broader activities and culture of organizations, relatively few studies
examine the effect of ECB efforts on organizations (MacDonald et al., 2003; Robinson &
Cousins, 2004). Despite concerns about the sustainability of evaluation capacity over
time, very few studies have examined whether or how capacity has been sustained
beyond the end of ECB initiatives, and those who have followed up have not done so
more than a year after the fact.
While it is important to be aware of the limitations described here, the studies
reviewed here represent the state of the art of what is known about evaluation capacity
building. They are valuable in helping to clarify what work people in the field are
classifying as ECB, what approaches seem to be successful or acceptable to the
organizations and individuals receiving capacity building interventions, and what
challenges and difficulties have been identified in ECB. Each of these themes is
discussed in more detail below. Table 1 provides further information on each of the
studies examined.
Strategies used for ECB.
A wide variety of different strategies have been identified in the available
literature on ECB. These strategies can be conceptualized in a rough continuum from less
intense evaluator involvement (e.g. group interventions, evaluator is less involved in
carrying out evaluation) to more intense involvement (more one-on-one involvement,
evaluator has greater responsibility for carrying out evaluation activities).
-
8/3/2019 Duffy ECB Working Paper
21/68
Evaluation Capacity 21
A commonly identified capacity building strategy was to provide participants with
written materials about evaluation. Manuals were the most frequently mentioned type of
material (Brandon & Higa, 2004; Campbell et al., 2004; Chinman et al., 1996; Keener &
Snell-Johns, 2006; Kirsh et al., 2005; & World Bank, 2004). Other materials provided
included evaluation resource lists or resource libraries, evaluation websites, and
evaluation newsletters (Kirsh, 2005; Schnoes et al., 2000). Stevenson et al. (2002) also
identified examples of high quality evaluation plans and shared those exemplary reports
with the organizations they were training. These materials might be a part of training, but
most commonly would be used by individuals working independently.
Many authors reported using workshops or other types of group trainings to
increase participants knowledge and skills about evaluation (Arnold, 2006; Brandon &
Higa, 2003; Brown et al., 2005; Campbell et al., 2004; Chinman et al., 1996; Compton et
al., 2002; Harper et al., 2003; Keener et al., 2005; Kirsh et al., 2005; Lennie et al., 2004;
MacDonald et al., 2003; Nagao et al., 2005; OSullivan & DAgostino, 2002; Stevenson
et al., 2002; Valery & Shakir, 2005; Woodland & Hind, 2002; World Bank, 2004).
Topics of training that were identified included information on the basics of doing
program evaluation and creating program logic models, workshops focused on planning
evaluations, and workshops on analyzing evaluation results. Perhaps the most ambitious
training was a four day training program designed by Nagao et al. to provide educators
(with little to no prior experience conducting evaluation) with the all skills necessary to
plan and conduct self-evaluation of their schools.
It is noteworthy that several researchers report that planned trainings on
evaluation topics were not carried out. King (2002) reported that although she had
-
8/3/2019 Duffy ECB Working Paper
22/68
Evaluation Capacity 22
initially envisioned holding workshops for staff of the school district where she was
working to build evaluation capacity, this plan was aborted because of the difficulty of
securing the necessary time, and the fact that such workshops were seen as a lower
priority for the district. Schnoes et al. (2000) also report that while they offered to
provide training on evaluation methods to three comprehensive community-initiatives
with whom they were conducting an empowerment evaluation, trainings did not take
place due to lack of time and the relatively low priority placed on formal evaluation by
participants.
Arnold (2006) raised concerns about the effectiveness of group trainings on
evaluation concepts for increasing evaluation capacity. Based upon her experiences with
providing training on logic models to 4-H educators (with little previous experience in
evaluation), she notes that the information provided in group trainings was rarely applied
when the educators returned to their regular work settings. She suggests that more
intensive one-on-one training or coaching may be necessary to facilitate mastery of
evaluation skills. This fits with findings of a review of the literature on implementation of
innovations in the field, which suggests that training alone is often not sufficient to
promote transfer of learning into practice (Fixsen et al., 2005).
The provision of technical assistance or consultation on evaluation issues is
another strategy identified in many of the studies of ECB reviewed here (Arnold, 2006;
Brandon & Higa, 2004; Campbell et al., 2004; Chinman et al., 1996; Keener & Snell-
Johns, 2006; MacDonald 2003; OSullivan & DAgostino, 2002; Schnoes et al., 2000;
Stevenson et al., 2002; Valery & Shakir, 2005; Woodland & Hind, 2002). Technical
assistance provides an opportunity to get individualized assistance with questions or
-
8/3/2019 Duffy ECB Working Paper
23/68
Evaluation Capacity 23
concerns related to the use of evaluation in practice. Brandon & Higa tracked the topics
for which technical assistance was requested, and report that they had many requests for
help identifying data collection methods (72% of the projects they worked with),
identifying the focus of their evaluation (56%), and recommendations related to data
analyses (56%). Authors of one study which did not include technical assistance
following training report that many respondents requested assistance when a six-month
follow-up was conducted (Nagao et al., 2003).
Many authors reported that they worked collaboratively with organizations to
carry out evaluation activities as a means of building capacity (Arnold, 2006; Biott &
Cook, 2000; Brandon & Higa, 2004; Compton et al, 2002; Fetterman, 2005; Keener &
Snell-Johns, 2006; King, 2002; Lentz et al., 2005; Harper et al., 2003; Robinson &
Cousins, 2004; Suarez-Balcazar et al. 2003; & Valery & Shakir, 2005). It is suggested
that the action of participating in evaluation builds evaluation capacity. Specific areas
where collaborative processes were noted include the helping organizations to take
stock of their programs and identify their expectations for the program, developing logic
models, developing evaluation plans, and developing data collection tools. Often the
collaborative process was not clearly described, and it is not always possible to tell how
deeply involved organization staff are. Several authors report this process is facilitated
through regular meetings (in person or by phone) devoted to discussing evaluation issues.
Others recommended that teachable moments be identified and used throughout the
evaluation process. Schnoes et al. (2000) note that they ended up taking a more directive
role in evaluation processes because of a lack time and motivation for involvement
among program staff. One author clearly articulated a vision of participation in
-
8/3/2019 Duffy ECB Working Paper
24/68
Evaluation Capacity 24
evaluation activities as the curriculum of ECB efforts, where the participants were
students of evaluation as well as participants (King, 2002). Arnold (2006) described an
intentional process of evaluation coaching by which assistance with evaluation is
gradually scaled back as participants gain confidence. However, she also notes that this is
possible of the availability of a full time staff member dedicated to ECB activities, which
is a resource that few programs have.
The strategies described above largely focus on the development of evaluation
capacity at the individual level. A number of strategies are also mentioned that might be
conceptualized more as organization-level strategies. One such strategy is the
development of an evaluation team or expertise within the organization (Compton et al.,
2002; GAO, 2003; King, 2002; Lennie et al., 2004; Robinson & Cousins, 2004; Valery &
Shakir, 2005, Woodland & Hind, 2002). These teams vary from informal groups working
on a specific evaluation activity together (e.g. carrying out focus groups) to more formal
groups which take on a greater role in conducting evaluation within the organization.
Other organizational interventions include making evaluation a part of staff job
descriptions (Brown et al., 2005), development of evaluation guidance documents
(Milstein et al., 2002), building leadership support for evaluation (GAO, 2003; World
Bank, 2004), and developing networks or other resources for identifying outside experts
to provide specialized guidance with challenging evaluation issues (Compton et al., 2002;
GAO, 2003; World Bank, 2004).
A wide variety of ECB strategies have been identified in the literature reviewed.
The majority of these are primarily focused on building individual-level evaluation
capacity. Another distinction can be made between activities like that provide less intense
-
8/3/2019 Duffy ECB Working Paper
25/68
Evaluation Capacity 25
evaluator involvement in the evaluation process, and those which require more intense
involvement on the part of evaluators. A full list of the strategies identified and the
authors that mentioned them is included in Table 2.
Outcomes of ECB.
As noted above, the nature ECB work and challenges with measuring evaluation
capacity make assessing the outcomes of ECB challenging. Somewhat ironically, the
methods used in most studies on ECB make attributing outcomes to ECB strategies
problematic. The results of studies of ECB which provide some empirical evidence of the
outcomes identified are described below. These results are grouped by the type of
evidence used. Outcomes which are based only on the authors report are not included in
this section. However, a list of the ECB outcomes identified in the studies reviewed, the
authors who reported each outcome, and the evidence provided for that outcome is
included in Table 3.
Several studies have examined the results of ECB using pre-post surveys.
Chinman et al. (2006) compared changes in knowledge, attitude, and skills related to
planning and evaluation of coalition members who participated in a capacity building
program with coalition members working on programs that were not a part of this effort.
They did not find a significant difference across the two groups, but secondary analyses
within the treatment group showed that individuals who had participated more in
evaluation activities showed greater increases in knowledge, attitude, and skills than
those who participated less. Stevenson et al. (2002) administered pre- and post-surveys to
participants in three different ECB workshops. They report that participants confidence
in their ability to use evaluation skills increased following their participation in the
-
8/3/2019 Duffy ECB Working Paper
26/68
Evaluation Capacity 26
workshops, but in some cases this confidence had dropped slightly by the time they
participated in the next workshop. Comparing surveys administered at the beginning and
end of a year long ECB project, Brandon and Higa (2004) also reported increases in
confidence in ability to do evaluation activities, but reported no significant change in
attitudes towards evaluation. However, they note that low participation rates make
interpreting the results problematic, since those who completed the post-survey tended to
be those who persisted in ECB, not those who had dropped out the efforts.
Three studies examined ECB outcomes using retrospective pre-post surveys.
These surveys asked respondents to assess what they knew about evaluation prior to the
ECB effort they participated in, and what they currently knew. In one case (Brown &
Reed, 2002) this comparison was from the beginning to end of a day long workshop. In
two other studies, the start of the ECB work was several years prior to completion of the
survey (Arnold, 2006; Keener & Snell-Johns, 2006). While this type of survey avoids the
problem of respondents overestimating their knowledge or ability before receiving
training (i.e. they dont know what they dont know), and is convenient because it does
not require baseline data collection, it may be problematic to ask people to accurately
assess their levels of knowledge or ability several years after the fact. Arnold (2006)
reports that ECB participants (most of whom had been involved in the effort for five
years) reported increased own evaluation skills and increased use of evaluation. Keener
and Snell-Johns (2006) also report increases in confidence in evaluation knowledge and
skills over the period when ECB took place (mean time of involvement at the time of the
assessment was 13.8 months). They also note that capacity specific to the program being
worked on increased more than did more general evaluation capacity. Participants in a
-
8/3/2019 Duffy ECB Working Paper
27/68
Evaluation Capacity 27
one-day evaluation training workshop reported increases in knowledge of key areas at the
end of the workshop (Brown & Reed, 2002).
Several authors have explored the effects of ECB using in depth qualitative
analyses of field notes, interviews, and focus groups (Lennie et al., 2004; Robinson &
Cousins, 2004). Lennie et al. examined the empowering and disempowering effects of
ECB on individuals who were highly involved in one of two community wide ECB
initiatives. It should be noted that this is the only study reviewed to explicitly mention
looking for possible harmful effects of ECB efforts. Lennie at al. report empowering
effects including: increased knowledge of planning and evaluation, increased evaluation
and communication skills, and the application of knowledge, skills and information. They
also report a number of disempowering effects for a small number of participants,
including: insufficient knowledge sometimes led to confusion (especially at first), lack of
technical resources, lack of influence/voice in the process, and frustration/lack of
confidence among some participants. Robinson and Cousins used qualitative methods to
examine the impact of long term collaborative evaluation on organizational learning.
They report that the participating organization demonstrated increased levels of
organizational learning over the course of the evaluation, including greater organizational
memory, an increase in the shared understanding of the program, and a greater amount of
double loop learning.
Much of the research on ECB outcomes is based on participants self reports.
However, some studies have examined ratings by external observers (typically the
evaluator or researcher) to provide external of assessment of ECB at the program or
organizational level (Chinman et al., 2006; Campbell et al., 2004; OSullivan &
-
8/3/2019 Duffy ECB Working Paper
28/68
Evaluation Capacity 28
DAgostino, 2002; & Stevenson et al., 2002). Chinman et al. rated quality of
implementation of planning and evaluation activities using a innovation-configuration
map designed for the components of the evaluation framework used. The small number
of programs involved limited their analyses, but based on a descriptive analysis, they
report that programs that received the intervention showed increases in capacity over
time, while comparison programs did not show an increase. Campbell et al. compared
evaluation reports submitted by the 10 sexual violence prevention programs they worked
with prior to the start of ECB and after ECB work was completed. Content analysis of the
10 reports showed significant increases in several areas, including the number of
programs using more sophisticated evaluation methods, collecting and analyzing their
own data, and using evaluation data for program improvement. OSullivan and
DAgostino (2002) also rated the evaluation reports or agencies they worked with to
build evaluation capacity. They reported an increase in evaluation quality, as the number
of programs reporting only service units or data-free outcomes decreased; and the
number of programs reporting outcomes supported by data increased. Stevenson et al.
rated the organizations that they worked with on their completion of 13 evaluation steps
prior to starting ECB and one year later. They report significant increases in the number
of agencies completing 9 of the 13 steps at the second measurement (including increases
in the organizational resources for evaluation, completion of preliminary evaluation steps,
conducting data collection, and planning for using evaluation results). They also noted
that organization-level improvement was positively correlated with the amount of
technical assistance received by these agencies.
-
8/3/2019 Duffy ECB Working Paper
29/68
Evaluation Capacity 29
Relatively few studies included information about whether evaluation capacity
was sustained following the ECB intervention, but those who did used follow-up
interviews with participants. Brown and Reed (2002) report that at a 6-month follow-up
following an evaluation training workshop, respondents were more likely to report that
they or someone else at their organization was carrying out various evaluation activities
than they had been prior to the workshop. Brown and Reed also noted that respondents
who reported a supportive organizational culture were more likely to have made progress
in implementing an evaluation system than those who reported the culture was not
supportive. Campbell et al. (2004) report that the majority of organizations that remained
open at the 1-year follow-up survey following ECB activities were continuing to conduct
evaluations (and organizations who were required to do evaluation were more likely to be
continuing to do so). Nagao et al. (2005) also report that most workshop participants were
working to implement evaluation in their schools at a 6-month follow-up interview.
Nagao et al. also noted that at this interview, many respondents identified a need for
consultation or technical assistance with some aspects of their evaluations.
To summarize across an extremely varied group of studies, the limited empirical
findings related to ECB outcomes suggest that efforts to increase evaluation capacity
have met with some success when evaluation capacity is measured at the individual and
organizational levels. Methodological limitations of most of these studies make it
difficult to attribute a causal relationship between ECB efforts and changes in evaluation
capacity; most are more appropriately considered outcome monitoring than outcome
evaluation (one exception is the work by Chinman et al. (2006), which uses a quasi-
experimental design). In addition, the nature of the ECB studies described here make
-
8/3/2019 Duffy ECB Working Paper
30/68
Evaluation Capacity 30
impossible to disentangle which ECB strategies had the greatest impact on the
development of evaluation capacity. Despite these limitations in the evidence base, a
number of findings have interesting implications. Findings from Chinman et al. (2006)
suggest that actual participation in planning and evaluation activities is tied to increases
in capacity in these areas. Likewise, Keener and Snell-Johns (2006) research suggests
areas of evaluation capacity directly associated with the work being done (i.e. program
specific evaluation capacity) increases more than more general evaluation capacity. Both
these findings are suggestive of support for the practice knowledge that learning by doing
evaluation is a key pathway to evaluation capacity. It is also noteworthy that very few
harmful outcomes of ECB were reported, although the only study that did was also the
only study which explicitly mentioned looking for them (Lennie et al., 2004).
Challenges to ECB.
Limited organizational resources necessary for carrying out evaluation
(particularly staff time) was the most commonly identified barrier to ECB (Brown et al.,
2005; Campbell et al., 2004; Chinman et al., 1996; King, 2002; Kirsh et al, 2005; Lennie
et al., 2004; Stevenson et al. 2002; Schnoes et al., 2000; Woodland & Hind, 2002). A
related concern was the difficulty of maintaining participation in ECB over time,
particularly in organizations and fields with very high staff turnover (Brandon & Higa,
2004; Brown et al., 2005; Chinman et al., 1996; Compton et al., 2002; Lennie et al.,
2004; Milstein et al., 2002; Stevenson et al. 2002; & Suarez-Balcazar et al., 2003).
While not a challenge per se, the need for ongoing guidance or technical
assistance in order to be able to carry out evaluation was noted by several authors
(Arnold, 2006; Campbell et al., 2004; Chinman et al., 1996, Nagao et al., 2005; &
-
8/3/2019 Duffy ECB Working Paper
31/68
Evaluation Capacity 31
Stevenson et al., 2002). Several expressed concern that the ready availability of technical
assistance might reduce the motivation of organization staff to develop technical
expertise of their own, and identified the slippery slope that exists between providing
technical assistance and doing it for them (Chinman et al., 1996; Stevenson et al.,
2002). Other challenges identified by multiple studies include: difficulties identifying
appropriate data collection tools and analyzing data, fear of evaluation, confusion among
participants about evaluation concepts, and lack of proven methods for ECB. Table 4
provides a listing of challenges to ECB identified in different studies.
Conclusions
Bringing Together ECB Theory and Existing Empirical Research
There are a number of areas where the existing research and theory related to
ECB are congruent, but also some areas where there are disconnects between the theory
and research. One area of disconnect is that the definitions of ECB, mainstreaming
evaluation, and evaluative inquiry all make central the idea of making evaluation a part of
the regular work of organizations. This idea of institutionalizing evaluation is central to
the theory of ECB, but to date is largely missing from the research on ECB. Relatively
few strategies were identified in the literature for bringing about a state of affairs in
which quality program evaluation and its appropriate uses are ordinary and ongoing
practices (Stockdill et al., 2002, p. 8). While several case studies identified some
indications that evaluation had been institutionalized to some degree (e.g. MacDonald et
al. (2003) described how evaluation gradually moved from a special occurrence to a
mandated part of all new programs), none of the more methodologically rigorous studies
examined this issue.
-
8/3/2019 Duffy ECB Working Paper
32/68
Evaluation Capacity 32
Another area of disconnect between ECB theory and research is that evaluation
theory (particularly drawing from international development) has focused on the
importance of cultivating demand for evaluation and organizational cultures supportive of
evaluation. The empirical studies reviewed here offered relatively few strategies aimed at
helping to develop such a culture. One useful recommendation that comes from
MacDonald (2003) is to incorporate evaluation gradually into the organization, so that
people have an opportunity to see how it works before they are asked to implement it.
This recommendation is congruent with Owens model of how to develop evaluation
culture theory. Several of the case studies reviewed here mention increased prominence
of or value placed on evaluation as a result of ECB activities (Arnold, 2006; Compton et
al., 2002; Lentz et al., 2005; MacDonald et al., 2003; Valery & Shakir, 2005) but only
MacDonald and colleagues present evidence of this. It is possible that work done to
increase the support (and demand) for evaluation within organizational culture might
reduce some of the commonly identified barriers to ECB work (the lack of organizational
resources for evaluation and difficulty maintaining staff participation).
A third area of potential disconnect between evaluation research and theory has to
do with the distinction (made most strongly by Stockdill et al. (2002)) between the
practice of carrying out evaluation and the practice of building evaluation capacity. In
practice, some of the case studies reviewed appeared to make little distinction between
carrying out a collaborative or participatory evaluation and building evaluation capacity.
To further complicate the picture, Patton (1997) has suggested that that participation in
evaluation can contribute to individual or organizational learning through process use,
and that the amount of participation in evaluation is likely to affect the amount of
-
8/3/2019 Duffy ECB Working Paper
33/68
Evaluation Capacity 33
learning that takes place. This theory is supported by findings from Chinman et al. (2006)
suggesting that those who participated in more evaluation activities built more capacity,
and from Keener & Snell-Johns (2006) that evaluation capacity directly relevant to the
activities in which individuals participated increased more than general evaluation
capacity. These findings suggest that providing opportunities to participate in evaluation
may be a powerful ECB tool. The point made by Stockdill et al. (2002) is well taken; just
carrying out an evaluation, even if it is collaborative or participatory, should not
necessarily be considered ECB. However, if evaluation participation is planned for in a
way that is mindful of the goal of evaluation capacity building and deliberately structures
participation in a way to maximize learning (e.g. King, 2002) then it makes sense to
classify it as a strategy for ECB (as well as the practice of evaluation).
Recommendations for Next Steps for the Field of ECB
It has been suggested that ECB is a field in its infancy (Stockdill et al., 2002;
Cousins et al., 2004) and this review of the research relevant to ECB does not contradict
that. Like an infant, the research literature on ECB is growing at a rapid pace (as is
suggested by the fact that the vast majority of the studies reviewed here were published
since the year 2000), but it still has a long way to go before reaching maturity. There is a
striking lack of research using strong methods (either quantitative or qualitative)
particularly given the complexity of the ECB process and the multi-level nature of the
work. While this is understandable given the newness of the field, MacKay emphasizes
the importance of ECB practitioners walking the walk by evaluating their own efforts,
and notes that this can be a strong tool for promoting the value of evaluation within an
organization.
-
8/3/2019 Duffy ECB Working Paper
34/68
Evaluation Capacity 34
The growing body of case studies focused on ECB experiences provides a useful
start for the development and refinement of ECB theory. However, while some provide
rich detail, these case studies are hampered in many instances by a lack of a systematic
way of compiling experiences learned, and by the limitations inherent in the fact that
most are from the perspective of the evaluator or ECB practitioner. It is crucial to learn
more about ECB from the perspective of those in the field working to build their own
evaluation capacity or not (e.g. Schnoes et al., 2000). Qualitative methods, such as
grounded theory, likely have much to offer here, particularly given the need to develop
and refine ECB theory, and the complex and multidimensional nature of the factors at
pay in ECB.
-
8/3/2019 Duffy ECB Working Paper
35/68
Evaluation Capacity 35
References
Arnold, M. E. (2006). Developing evaluation capacity in Extension 4-H Field Faculty: A
framework for success.American Journal of Evaluation, 27(2), 257-269.
Baizerman, M., Compton, D. W. & Stockdill, S. H. (2002a). Editors Notes.New
Directions for Evaluation, 93, 1-6.
Baizerman, M., Compton, D. W. & Stockdill, S. H. (2002b). New directions for ECB.
New Directions for Evaluation, 93, 109-119.
Baizerman, M., Compton, D. W. & Stockdill, S. H. (2002c). Summary and analysis of the
case studies: Themes across the cases.New Directions for Evaluation, 93, 101-
107.
Biott, C., & Cook, T. (2000). Local evaluation in a national early years excellence centres
pilot programme: Integrating performance management and participatory
evaluation.Evaluation, 6(4), 399-413.
Botcheva, L., White, C. R., & Huffman L. C. (2002). Learning culture and outcomes
measurement practices in community agencies.American Journal of Evaluation,
23(4), 421-434.
Boyle, R. (1999). Professionalizing the evaluation function: Human resource
development and the building of evaluation capacity. In R. Boyle & D. Lemaire
(eds.)Building Effective Evaluation Capacity, 135-151. New Brunswick, NJ:
Transaction Publishers.
Brandon, P. R., & Higa, T. A. (2004). An empirical study of building the evaluation
capacity of K-12 site managed project personnel. The Canadian Journal of
Program Evaluation, 19(1), 125-142.
-
8/3/2019 Duffy ECB Working Paper
36/68
Evaluation Capacity 36
Brown, N. L., Luna, V., Ramirez, M. H., Vail, K. A., & Williams, C. A. (2005).
Developing an effective intervention for IDU women: A harm reduction approach
to collaboration.AIDS Prevention and Education, 17(4), 317-333.
Brown, R. E., & Reed, C. S. (2002). An integral approach to evaluating outcome
evaluation training.American Journal of Evaluation, 23(1), 1-17.
Campbell, R., Dorey, H., Naegeli, M., Grubstein, L., Bennett, K., Bonter, F., et al.
(2004). An empowerment evaluation model for sexual assault programs:
Empirical evidence of effectiveness.American Journal of Community
Psychology, 34, 251-262.
Chinman, M., Hunter, S. B., Ebener, P., Paddock, S., Stillman, L., Imm, P., &
Wandersman, A. (2006). The Getting To Outcomes demonstration and
evaluation: An illustration of the Prevention Support System. Manuscript
submitted for publication.
Compton, D. W., Glover-Kudon, R., Smith, I. E., & Avery, M. E. (2002). Ongoing
capacity building in the American Cancer Society (ACS) 1995-2001.New
Directions for Evaluation, 93, 47-61.
Cousins, J. B., Goh, S. C., Clark, S., & Lee, L. E. (2004). Integrating evaluative inquiry
into the organizational culture: A review and synthesis of the knowledge base.
The Canadian Journal of Program Evaluation, 19(2), 99-141.
Duignan, P. (2003). Mainstreaming evaluation or building evaluation capability? Three
key elements.New Directions for Evaluation, 99, 7-21.
-
8/3/2019 Duffy ECB Working Paper
37/68
Evaluation Capacity 37
Fetterman, D. M. (2005). Empowerment evaluation: From the digital divide to academic
distress. In D. M. Fetterman & A. Wandersman (Eds.),Empowerment evaluation
principles in practice (pp. 92-122). New York: Guilford.
Forss, K., Rebien, C. C., & Carlsson, J. (2002). Process use of evaluations: Types of use
that precede lessons learned and feedback.Evaluation, 8(1), 29-45.
General Accounting Office. (2003). Program Evaluation: An Evaluation Culture and
Collaborative Partnerships Help Build Agency Capacity (GAO-03-454).
Washington, DC: General Accounting Office.
Grudens-Schuck, N. (2003). The rigidity and comfort of habits: A cultural and
philosophical analysis of the ups and downs of mainstreaming evaluation.New
Directions for Evaluation, 99, 23-32.
Harper, G. W., Contreras, R., Bangi, A., & Pedraza, A. (2003). Collaborative process
evaluation: Enhancing community relevance and cultural appropriateness in HIV
prevention.Journal of Prevention and Intervention in the Community, 26(2), 53-
69.
Keener, D. C. & Snell-Johns, J. (2006).Measuring Evaluation Capacity: Results from a
Pilot Administration of the South Carolina Evaluation Capacity Survey.
Unpublished manuscript, University of South Carolina.
Keener, D. C., Snell-Johns, J., Livet, M., & Wandersman, A. (2005). Lessons that
influenced the current conceptualization of Empowerment Evaluation: Reflections
from two evaluation projects. In D. M. Fetterman & A. Wandersman (Eds.),
Empowerment evaluation principles in practice (pp. 73-91). New York: Guilford.
-
8/3/2019 Duffy ECB Working Paper
38/68
Evaluation Capacity 38
King, J. A. (2002). Building the evaluation capacity of a school district.New Directions
for Evaluation, 93, 63-80.
Kirsh, B., Krupa, T., Horgan, S., Kelly, D., & Carr, S. (2005). Making it better: Building
evaluation capacity in community mental health. Psychiatric Rehabilitation
Journal, 28(3), 234-241.
Lennie, J. (2005). An evaluation capacity-building process for sustainable community IT
initiatives: Empowering and disempowering impacts.Evaluation, 11(4), 390-414.
Lennie, J., Hearn, G., Simpson, L., da Silva, E. K., Kimber, M., & Hanrahan, M. (2004).
Building community capacity in evaluating IT projects: Outcomes of the
LEARNERS project. Queensland, Australia: Creative Industries Research and
Applications Centre.
Lentz, B. E., Imm, P. S., Yost, J. B., Johnson, N. P., Barron, C., Lindberg, M. S., &
Treistman, J. (2005). Empowerment evaluation and organizational learning: A
case study of a community coalition designed to prevent child abuse and neglect.
In D. M. Fetterman & A. Wandersman (Eds.),Empowerment evaluation
principles in practice (pp. 155-182). New York: Guilford.
Mackay, K. (2002). The World Banks ECB experience.New Directions for Evaluation,
93, 81-99.
McDonald, B., Rogers, P., & Kefford, B. (2003) Teaching people to fish? Building
evaluation capability of public sector organizations.Evaluation, 9(1), 9-29.
Miller, T. I., Kobayashi, M. M., & Noble, P. M. (2006). Insourcing, not capacity
building, a better model for sustained program evaluation.American Journal of
Evaluation, 27(1), 83-94.
-
8/3/2019 Duffy ECB Working Paper
39/68
Evaluation Capacity 39
Milstein, B., Chapel, T. J., Wetterhall, S.F., & Cotton, D. A. (2002). Building capacity
for program evaluation at the Centers for Disease Control and Prevention.New
Directions for Evaluation, 93, 27-46.
Milstein, B., & Cotton, D. (2000, March).Defining Concepts for the Presidential Strand
on Building Evaluation Capacity. Working paper circulated in advance of the
November, 2000, meeting of the American Evaluation Association.
Nagao, M., Kuji-Shikatani, K., & Love, A. J. (2005). Preparing school evaluators:
Hiroshima pilot test of the Japan Evaluation Societys Accreditation Project. The
Canadian Journal of Program Evaluation, 20(2), 125-155.
OSullivan, R. G., & DAgostino, A. (2002). Promoting evaluation through collaboration:
Findings from community-based programs for young children and their families.
Evaluation, 8(3), 372-387.
Owen, J. M. (2003). Evaluation culture: A definition and analysis of its development
within organizations.Evaluation Journal of Australasia, 3(1), 43-47.
Patton, M. Q. (1997). Utilization-focused Evaluation. Thousand Oaks, CA: Sage
Publications, Inc.
Preskill, H., & Torres, R. T. (1999).Evaluative Inquiry for Learning in Organizations.
Thousand Oaks, CA: Sage Publications, Inc.
Preskill, H., & Torres, R. T. (2000). The learning dimension of evaluation use.New
Directions for Evaluation, 88, 25-37.
Preskill, H., Zuckerman, B., & Matthews, B. (2003). An exploratory study of process use:
Findings and implications for future research.American Journal of Evaluation,
24(4), 423-442.
-
8/3/2019 Duffy ECB Working Paper
40/68
Evaluation Capacity 40
Robinson, T. T., & Cousins, J. B. (2004). Internal participatory evaluation as
organizational learning: A longitudinal case study. Studies in Educational
Evaluation, 30, 1-22.
Sanders, J. R. (2003). Mainstreaming Evaluation.New Directions for Evaluation, 99, 3-6.
Schnoes, C. J., Murphy-Berman, V., & Chambers, J. M. (2000). Empowerment
evaluation applied: Experiences, analysis, and recommendations from a case
study.American Journal of Evaluation, 21(1), 53-64.
Stevenson, J. F., Florin, P., Mills, D. S., & Andrade, M. (2002).Building evaluation
capacity in human service organizations: A case study.Evaluation and Program
Planning, 25, 233-243.
Stockdill, S. H., Baizerman, M., & Compton, D. W. (2002). Toward a Definition of the
ECB Process: A Conversation with the ECB Literature.New Directions for
Evaluation, 93, 7-25.
Suarez-Balcazar, Y., Orellana-Damacela, L., Portillo, N., Sharma, A., & Lanum, M.
(2003). Implementing an outcomes model in the participatory evaluation of
community initiatives.Journal of Prevention and Intervention in the Community,
26(2), 5-20.
Torres, R. T., & Preskill, H. (2001). Evaluation and organizational learning: Past, present,
future.American Journal of Evaluation, 22(3), 387-395.
Trevisan, M. S. (2002). Evaluation capacity in K-12 school counseling programs.
American Journal of Evaluation, 23(3), 291-305.
-
8/3/2019 Duffy ECB Working Paper
41/68
Evaluation Capacity 41
Trochim, W. M. The Research Methods Knowledge Base, 2nd Edition. Internet WWW
page, at URL: (version current
as of January 16, 2005).
Toulemonde, J. (1999). Incentives, constraints, and culture-building as instruments for
the development of evaluation demand. In R. Boyle & D. Lemaire (eds.),
Building Effective Evaluation Capacity, (pp. 153-174), New Brunswick, NJ:
Transaction Publishers.
Toulemonde, J. (1995). The emergence of an evaluation profession in European
countries: the case of structural policies. Knowledge and Policy, 8(3), 43-54.
Valery, R., & Shakir, S. (2005). Evaluation capacity building and humanitarian
organization.Journal of Multidisciplinary Evaluation, 3, 78-112.
Woodland, J., & Hind, J. (2002). Capacity building evaluation of capacity building
programs. Paper presented at the 2002 meeting of the Australasian Evaluation
Society International Conference, October/November, 2002.
World Bank (2004). Evaluation Capacity Development: OED Self Evaluation.
Washington, DC: The World Bank Operations Evaluation Department.
-
8/3/2019 Duffy ECB Working Paper
42/68
Evaluation Capacit
Table 1
Characteristics of the Studies Reviewed
Study Sample Context Purpose/Questions Methods Used Key Arnold (2006) 4-H field
educators in thestate of Oregon(30 out of 37participated insurvey)
Evaluation capacitybuilding effort forfield coordinatorswho had little timeor training for doingplanning andevaluation
Examine changes tothe evaluationcapacity of 4-H fieldcoordinators followinga 5-year ECBprogram
Retrospectivepre-post survey,reflective casestudy
Quafromeducdevethe tincreprac
Biott & Cook(2000)
1 center for earlychildhooddevelopment in arural area ofnorth England
(center staff,parents and othercommunitymembersparticipated)
Local evaluationusing aparticipatoryevaluationframework; both a
local and nationalevaluation wererequired in order toreceive funding forthis pilot program
Describe the tensionbetween localparticipatoryevaluation focusedon local program
improvement incontext and nationalevaluation focusedon performancemanagement
Reflective casestudy
Impoindicfor eShobase
rolesstaffcapa
Brandon & Higa(2004)
Staff of 17projects across 2school districts inHawaii; 59individualsparticipated inmeetings andworkshops (41
respondentscompleted pre-survey, 37completed post-survey, only 20completed both)
The school districtrequired theprograms to beevaluated, andcontracted with theevaluators toprovide evaluationprofessional
development tothe program staffinvolved
Examine the effectsof professionaldevelopment onparticipantsevaluation capacity
Pre-post survey(limited by lowresponse rate,particularly onpost-survey)
Pre-conf(evamoreundesigndifficin ab
objeevalNo ceval
-
8/3/2019 Duffy ECB Working Paper
43/68
Evaluation Capacit
Study Sample Context Purpose/Questions Methods Used Key
Brown & Reed,2002
Participants inevaluationtrainingworkshops held
for local CBOsheld in Michigan(number ofparticipants notprovided)
United Waysponsoredvoluntary trainingson evaluation;
participants wereexpected totransfer/disseminate whatthey learned totheir organizations
Examine the extent towhich trainingparticipants reportedincreased knowledge,
skills, and thetransfer of knowledgeand skills to their jobs
Retrospectivepre-post surveyat training, 6-month follow up
survey
Mosundebeliefor th
perfowereThosenviless eval
Brown et al., 2005 4 HIV PreventionCBOs in southernCalifornia
Collaboration ofcounty, evaluationcontractor and 4CBOs to evaluatea new HIVprevention program
Illustrate how onegroup of agenciesovercame barriers tobuild capacity fordelivery andevaluation of theprogram
Reflective casestudy
Highresotrainchalevalby thwere
Campbell et al.(2004)
10 Rapepreventionprograms; 24rape victimsservicesprograms in thestate of Michigan
State funderswanted programsto evaluateprograms, andbrought together anevaluation team todevelop localevaluations
Assess theeffectiveness of anempowermentevaluation approachintended to buildevaluation capacity inlocal programs
Participantssatisfactionratings,comparison ofquality ofevaluationreports at thestart and end ofthe program, and1-year follow-upinterviews
Baseevalprevsignagenincluevalanalfindi100%90%(opewere
(eithoutc
-
8/3/2019 Duffy ECB Working Paper
44/68
-
8/3/2019 Duffy ECB Working Paper
45/68
Evaluation Capacit
Study Sample Context Purpose/Questions Methods Used Key
examined theirexperience anddeveloped casestudies
Harper et al.(2003)
1 HIV/AIDSservice CBO inChicago
Collaborativeempowermentevaluation;partnershipbetween CBO anduniversity faculty &students
Describe thedevelopment,implementation, anduse of processevaluation by a teamof communitymembers anduniversity faculty andstudents
Reflective casestudy
Succthe dcollenumThe baseyout
Keener & SnellJohns (2006);Keener et al.(2005)
2 communityinitiatives, 15staff membersparticipated in the
survey
Initiatives hired anevaluation team toconductempowerment
evaluation andbuild self-evaluation capacity
Examine whetherthere were changesin evaluation capacityfollowing ECB
Retrospectivepre-post survey(Keener & Snell-Johns) and
reflective casestudy (Keener etal.)
Survsignevaltheir
ECBevalto thon) ievalseenimpo(regathe sthe e
-
8/3/2019 Duffy ECB Working Paper
46/68
Evaluation Capacit
Study Sample Context Purpose/Questions Methods Used Key
King (2002) 1 school districtin Minnesota
Internal evaluatorhired to carry outevaluations andbuild capacity of
the staff to engagein evaluation anddata-baseddecision-making;high level ofsupport fromsuperintendent, butlimited resources
Describe the processof building evaluationcapacity, includingsuccesses and
challenges
Reflective casestudy
Comthoscoupsign
ECBevalvisibapprpartiand adm
Kirsh et al. (2005) Members of afederation ofcommunitymental healthagencies in
Ontario; 168people from 101agenciesparticipated inworkshops(representing54% of federationmembership)
The federationidentified a need forevaluation capacitydevelopment andentered a
partnership with 2universities to buildevaluation capacityamong members
Describe a province-wide ECB effortincluding successes,challenges, and waysto sustain capacity
development
Reflective casestudy & survey ofparticipants
97%dointhosmodrepo
provinitiainfornot peffec
-
8/3/2019 Duffy ECB Working Paper
47/68
Evaluation Capacit
Study Sample Context Purpose/Questions Methods Used Key
Lennie et al.(2004); Lennie(2005)
2 ruralcommunities inAustralia,including 5
collaboratingorganizations and15 coreparticipants whowere included inmost analyses
Initiative intendedto increasecommunityinvolvement in and
capacity forplanning andevaluating ITprojects.Participatory actionresearch involvingresearchers andcommunitymembers
Examinedempowering anddisempoweringeffects on individual
participants
Quantitative &qualitativemethodsincluding
questionnaires,focus groups,interviews, andfield notes
Lennincluplaneval
and skillsDiseinclusomtechinfluof coparti
Mackay (2002) ECB outreachactivities at theWorld Bank
The World Bankhas a long historyof involvement inevaluation, but
began ECB workmore recently
Describe ECBactivities of the WorldBank and how ECBbecame a valued
activity of the Bank
Reflective casestudy
Maclessothe WHe a
is shECB
McDonald et al.(2003)
1 Division withinstate governmentof Victoria,Australia
Increasing pressureon the Division foraccountability andoutcomes, coupledwith a shift in focusfrom many smallerprojects to fewerlarger ones
Identification ofsuccessful capacitybuilding strategies
Case study of 5-year ECBinitiative(including use ofqualitative andquantitativemeasures frominternalevaluation ofECB activities)
Partevalexistfuturare sorgamainactiv
-
8/3/2019 Duffy ECB Working Paper
48/68
-
8/3/2019 Duffy ECB Working Paper
49/68
E
valuationCapacity
49
Study
Sam
ple
Context
Purpose/Questions
MethodsUsed
KeyFindings
Robinson&
Cousins,2004
Asingle
organization(a
partnershipof
gov
ernment
age
nciesand
national
organizations)in
Can
ada
Participatory
evaluationofa
singleprogram
beingimplemente
d
bytheorganizatio
n
Empiricallyexamine
thelinksbetween
participatory
evaluationand
organizational
learning
Longitudinal
single
case
study,including
qualitative
analysisof28
interviews,focus
group
,participant
obser
vationand
reflec
tion.
Qualitativeanalysis
showed
increasedlevelsoforganizational
learning,includingg
reater
organizationalmem
ory,anincrease
inthesharedunderstandingofthe
program,andagreateramountof
doublelooplearnin
g
Schnoesetal.
(2000)
3Comprehensive
Com
munity
Initiatives(CCIs)
in3
communities
acrossthestate
ofN
ebraska
Thestateof
Nebraska
contractedwiththe
evaluatorsto
evaluatethe3CC
Is
andtheevaluators
optedtousean
empowerment
evaluation
approach
Describethe
challenges
experiencedasthe
evaluatorsattempted
tobuildcapacityas
partofan
empowerment
evaluation
Reflectivecase
study,surveyof
CCIp
rogram
coord
inatorsand
board
members
abouttheir
perce
ptionsof
theevaluation
effort
InvolvementofCCI
staffmembersin
evaluationwasmuc
hlessactivethan
anticipated,leading
totheevaluators
takingamoredirectiveapproach.
CCIstaffreportedth
attheywere
busyandevaluation
wasnota
priority.Asurveysh
owedthat50%o
f
CCIstaffandboard
members
preferredtheevalua
torsactas
traditionalexternale
valuators.There
werevariableviews
ofthenature
andpurposeofevaluation.
Stevensonetal.
(2002)
13CBOscarrying
out
substance
use
prevention
(inc
luding25
participants
across3different
wor
kshops)
Evaluatorshiredb
y
thestatetobuild
capacityofCBOs
toevaluationtheir
preventionefforts.
Examinetheeffects
ofcapacitybuilding
efforts
Individuallevel
capac
ity
meas
uredwith
pre-posttraining
questionnaires
Agencylevel
capac
ity
meas
uredwith
evaluationstaff
rating
sofCBO
comp
letionof
evaluationsteps
Surveysshowedinc
reasedlevelsof
confidenceinability
tocarryout
evaluationactivities,withjumpsin
confidenceoccurringfollowingthe
trainingdeali