Research on Evaluation Use - Western Michigan University

Research on Evaluation Use

A Review of the Empirical LiteratureFrom 1986 to 2005

Kelli Johnson

Lija O. Greenseid

Stacie A. Toal

Jean A. King

Frances Lawrenz

University of Minnesota

Boris Volkov

University of North Dakota

This paper reviews empirical research on the use of evaluation from 1986 to 2005 using Cousins

and Leithwood’s 1986 framework for categorizing empirical studies of evaluation use conducted

since that time. The literature review located 41 empirical studies of evaluation use conducted

between 1986 and 2005 that met minimum quality standards. The Cousins and Leithwood

framework allowed a comparison over time. After initially grouping these studies according

to Cousins and Leithwood’s two categories and twelve characteristics, one additional

category and one new characteristic were added to their framework. The new category is

stakeholder involvement, and the new characteristic is evaluator competence (under the

category of evaluation implementation). Findings point to the importance of stakeholder

involvement in facilitating evaluation use and suggest that engagement, interaction, and

communication between evaluation clients and evaluators is critical to the meaningful use of

evaluations.

Keywords: evaluation use; evaluation influence; stakeholder involvement; literature review;

research on evaluation

In recent years, scholars have advanced calls for research on program evaluation and

especially on the impact of evaluations (e.g., Henry & Mark, 2003b; Scriven, 2007). As

Henry and Mark state, there is ‘‘a serious shortage of rigorous, systematic evidence that can

guide evaluation or that evaluators can use for self-reflection or for improving their next

evaluation’’ (2003b, p. 69). A time-honored method for providing guidance entails synthesiz-

ing existing research to identify what is known about evaluations and what remains to be

investigated. This is the approach taken in the current review of evaluation use, one of the few

Authors’ Note: This material is based on work supported by the National Science Foundation under Grant No. REC

0438545. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the

authors and do not necessarily reflect the views of the National Science Foundation. The authors gratefully

acknowledge Stuart Appelbaum for his contributions to this article. Correspondence concerning this article should

be addressed to Kelli Johnson, University of Minnesota, 2221 University Avenue SE, Suite 345, Minneapolis, MN

55414; phone: þ1 (612) 624-1457; e-mail: [email protected].

American Journal of

Evaluation

Volume 30 Number 3

September 2009 377-410

# 2009 American Evaluation

Association

10.1177/1098214009341660

http://aje.sagepub.com

hosted at

http://online.sagepub.com

377

at WESTERN MICHIGAN UNIVERSITY on June 29, 2010 http://aje.sagepub.comDownloaded from


topics in evaluation on which numerous empirical studies exist. Christie (2007, p. 8) notes,

‘‘Evaluation utilization is arguably the most researched area of evaluation and it also receives

substantial attention in the theoretical literature.’’ We define evaluation use or utilization—

evaluation scholars use the terms interchangeably—as the application of evaluation processes,

products, or findings to produce an effect.

Since the 1970s, naming the types of evaluation use has been the subject of continuing dis-

cussion. In reviewing this discussion to date, Alkin and Taut (2003) label two distinct aspects

of use: process use and use of evaluation findings. Process use is the newer concept, defined by

Patton (1997, p. 90) as ‘‘individual changes in thinking and behavior and program or organiza-

tional changes in procedures and culture that occur among those involved in evaluation as

a result of the learning that occurs during the evaluation process.’’ The use of findings is

traditionally divided into three types: instrumental, conceptual, or symbolic (King & Pechman,

1984; Leviton & Hughes, 1981). Instrumental use refers to instances where someone has used

evaluation knowledge directly. Conceptual use refers to cases when no direct action has been

taken but where people’s understanding has been affected. Symbolic use refers to examples

when a person uses the mere existence of the evaluation, rather than any aspect of its results,

to persuade or to convince.

Moving beyond the first quarter century of use research, the new millennium has witnessed

theoretical activity that has reconceptualized the field’s understanding of its impact. Scholars

now view evaluations as having intangible influence on individuals, programs, and commu-

nities. Focusing solely on the direct use of either evaluation results or processes has not

adequately captured broader level influences (Alkin & Taut, 2003; Henry & Mark, 2003a,

2003b; Kirkhart, 2000; Mark & Henry, 2004). What has potentially emerged from this activity

is a more nuanced understanding of evaluation’s consequences using evaluation influence as a

unifying construct. Kirkhart’s ‘‘integrated theory’’ defines influence as ‘‘the capacity or power

of persons or things to produce effects on others by intangible or indirect means’’ (2000, p. 7).

Kirkhart envisions three dimensions of evaluation influence, represented as a cube-like figure:

source (evaluation process or results), intention (intended or unintended), and time (immedi-

ate, end-of-cycle, long-term).

Mark and Henry (Henry & Mark, 2003a, 2003b; Mark & Henry, 2004; Mark, Henry, &

Julnes, 1999) have also pushed for broadening the way evaluators conceptualize the conse-

quences of their work. They argue that the goal of evaluation is social betterment and suggest

the need to identify the mechanisms through which evaluations lead to this ultimate goal along

differing paths of influence and at different levels (i.e., individual, interpersonal, and collec-

tive). Mark and Henry map out a logic model for evaluation, focusing on evaluation conse-

quences related to the improvement of social conditions. Just as program theory connects

program activities with outcomes while also explaining the processes through which the

outcomes are achieved, program theory of evaluation by Mark and Henry identifies evaluation

as an intervention with social betterment as its ultimate outcome. They label traditional notions

of instrumental, conceptual, and persuasive use more specifically as, for example, skill acqui-

sition, persuasion, or standard setting. These, then, would be the mechanisms through which

social betterment can be achieved.

Building on these ideas, Alkin and Taut (2003) carefully distinguish between evaluation

use and influence. To them, evaluation use ‘‘refers to the way in which an evaluation and

information from the evaluation impacts the program that is being evaluated’’ (Alkin &

Taut, 2003, p. 1). In their view, evaluators are aware of these evaluation impacts, both

intended and unintended. By contrast, ‘‘the concept of influence adds to the concept of use

in instances in which an evaluation has unaware/unintended impacts’’ (p. 9, emphasis in

original).

378 American Journal of Evaluation / September 2009



Structuring the Present Review

In structuring this literature review, we considered several options. Cousins (2003) draws a

logic model for program evaluation that builds on the knowledge utilization literature, but its

focus on participatory evaluation made it inappropriate for a review of evaluation use research.

Cousins, Goh, Clark, and Lee (2004) present a comprehensive framework of evaluative

inquiry as an organizational learning system, but, again, it includes many concepts other than

evaluation use.

Given the emergence of influence as a construct, another possibility was to apply the new

concept to analyze the existing literature. This proved impractical for three reasons. First,

some of the research we reviewed was conducted before Kirkhart’s (2000) work was pub-

lished. Second, given the newness of the term, there was little empirical research on influence,

although we did include it in our searches. Indeed, even studies conducted in the 5 years since

the term emerged (2000–2005) did not necessarily examine evaluation influence; moreover,

examining use through the lens of influence was not necessarily helpful because influence

is indirect and we were examining direct use. Third, and perhaps most important, the concept

of influence presented in Henry and Mark (2003a, 2003b) and Mark and Henry (2004) was not

defined, and the discussion of pathways, processes, and mechanisms did not provide sufficient

clarity to structure the review (Nunneley, 2008; Weiss, Murphy-Graham, & Birkeland, 2005).

We decided, therefore, to use the seminal study that Cousins and Leithwood conducted in

1986—one of the most ambitious and rigorous reviews of empirical research on evaluation use

ever conducted—as the underlying structure for this review, as well as more recent work by

Shulha and Cousins (1997). Although Cousins’ own conceptualizations of the topic have

evolved since this point, the taxonomy of evaluation use presented in the 1986 model was the

most comprehensive, well defined, and concrete.

Cousins and Leithwood Framework

Cousins and Leithwood (1986) identified 65 empirical studies of evaluation use conducted

between 1971 and 1985 through computerized searches of keywords including ‘‘evaluation

utilization,’’ ‘‘data use,’’ ‘‘decision making,’’ and ‘‘knowledge utilization.’’ They supplemen-

ted this process with manual searches of relevant journals and other literature reviews. After

establishing their sample, Cousins and Leithwood coded each study according to its orienta-

tion toward dependent variables (i.e., the type of use examined: use as decision making, use

as education, use as the processing of information, or ‘‘potential’’ use) and its orientation

toward independent variables.

The aspects of evaluation use examined in the 65 empirical studies were clustered into two

categories of factors related to evaluation use: (a) characteristics of evaluation implementa-

tion, and (b) characteristics of the decision or policy setting. Each of these categories contained

six characteristics. The six evaluation implementation characteristics were (a) evaluation

quality, (b) credibility, (c) relevance, (d) communication quality, (e) findings, and (f) timeli-

ness. The six decision- or policy-setting characteristics were (a) information needs, (b) decision

characteristics, (c) political climate, (d) competing information, (e) personal characteristics,

and (f) commitment or receptiveness to evaluation. Using a ‘‘prevalence of relationship’’

index, Cousins and Leithwood (1986) identified evaluation quality as the most important char-

acteristic, followed by decision characteristics, receptiveness to evaluation, findings, and

relevance.

Johnson et al. / Research on Evaluation Use 379



Shulha and Cousins (1997) described developments that had occurred since the review by

Cousins and Leithwood, including the following:

The rise of considerations of context as critical to understanding and explaining use; identification

of process use as a significant consequence of evaluation activity; expansion of conceptions of use

from the individual to the organization level; and diversification of the role of the evaluator to

facilitator, planner and educator/trainer (p. 195).

The present review incorporates these developments as well.

Importantly, these two major reviews of the use literature (Cousins & Leithwood, 1986;

Shulha & Cousins, 1997) differ in that the first considered only empirical research whereas

the more recent included theoretical or reflective case narratives in addition to empirical stud-

ies. Yet, many potentially instructive studies were excluded from the 1997 review, either

because they were conducted as doctoral dissertations or because they were not published

in journals. Neither review took into account the quality of the evidence gathered in the indi-

vidual studies when synthesizing the results. Consequently, the findings from studies in which

there could be serious methodological flaws potentially were presented alongside higher qual-

ity, rigorously conducted studies. To rectify these concerns, the current review included

empirical studies of evaluation use; examined journal articles, dissertations, reports, and book

chapters; and screened each study according to a predetermined set of criteria related to meth-

odological quality. In this review, we employ the term ‘‘use’’ rather than ‘‘influence,’’

although we view use broadly. We attempt to identify it as ‘‘process use’’ or ‘‘use of findings’’

and classify it as instrumental, conceptual, or symbolic.

Method

The research team collected relevant publications by conducting electronic searches for the

terms ‘‘evaluation utilization,’’ ‘‘evaluation use,’’ and ‘‘evaluation influence’’ in PsycINFO,

Education Resources Information Center (ERIC), Education (Sage), Social Services Abstracts,

Sociological Abstracts, and Digital Dissertations in keywords, titles, descriptors, and abstracts.

Additionally, the team consulted other published literature reviews, including Hofstetter and

Alkin (2003). Finally, the team conducted a manual review (looking for relevant research

based on titles) of the following evaluation-related journals: American Journal of Evaluation,

Canadian Journal of Program Evaluation, Evaluation, Evaluation Practice, Evaluation

and Program Planning, Evaluation Review, New Directions for Evaluation, and Studies in

Educational Evaluation. The searches examined only the literature written in English,

although the authors did not exclude research conducted outside the United States.

The searches returned over 600 journal articles, reports, and book chapters and 48 disserta-

tions. After scanning publication titles and abstracts, the team eliminated clearly irrelevant

publications. Then, the team closely reviewed 321 abstracts to assess whether the publication

met the following criteria: (a) an empirical research study (to be considered an empirical study

the article had to present information about the data collection methods used to inform the

claims made); (b) a focus on program or policy evaluation or needs assessment (not personnel

evaluation, accountability/student assessment studies, data-driven decision making, etc.); (c) a

published journal article, book, publicly accessible evaluation report, or dissertation (not a

conference presentation or other nonpublished work); (d) the inclusion of evaluation use or

influence as at least one of the variables under study; and (e) a publication date between

January 1, 1986 and December 31, 2005.




After the abstract review, the team identified 98 publications that warranted a full-text

review; these were subsequently screened again on all five criteria. This process yielded 47

articles that initially comprised the basis for this analysis. At least two trained screeners cate-

gorized and critiqued each study using a standardized review form developed and refined with

the input of several evaluation experts.

The rating form contained questions about each study’s methodology, choice of theory,

operationalization of dependent variable (measures of use), and independent variables (char-

acteristics affecting use). In addition, as noted, a quality rating was assigned to each study. The

quality rating was based on criteria adapted from Guarino, Santibanez, Daley, and Brewer

(2004) and Guba and Lincoln (1989). It considered aspects such as the clarity of the problem

statement, soundness of research design, strength of the link between evidence presented and

conclusions, and the extent to which bias was addressed. The team also assessed the sample

size and selection, measurement of variables, and statistical interpretation of the quantitative

studies, as well as the methodological appropriateness, transparency, descriptive richness, and

statement of researcher biases of the qualitative studies.

The reviewers independently assigned quality ratings across five levels: poor, adequate-low,

adequate-solid, adequate-high, and excellent. If the screeners did not agree on any particular

aspect of the review, the article was brought to a team meeting during which it was discussed

and consensus among the six researchers was reached regarding the rating. Although this

consensus-driven process for reviewing and assigning quality ratings was time-consuming, the

resulting judgments represent the agreement of two professors of evaluation and then four

evaluation doctoral students. We believe our process was both representative and fair.

On completion of this in-depth screening process, 41 of 47 studies (87.2%) were found to be

adequate or above. Six of the studies (12.8%) were rated as poor and eliminated from our sam-

ple. These ‘‘poor’’ studies suffered from a cursory description of the methods, weak sampling

or data analysis methods, poor measurements of use (e.g., not providing definitions of use and/

or using only one question as a measure of use), poorly supported generalizations, and/or

inadequate attention to likely researcher biases. The 41 studies that exceeded the minimum

quality criteria were used in the analyses presented below and are described in detail in the

Appendix. Six of these studies were published outside the United States.

Findings

Findings from the 41 studies are presented following the framework of 2 categories—

evaluation implementation and decision or policy setting—and 12 characteristics by Cousins

and Leithwood (1986). These two categories were helpful for organizing the majority of the

studies found in this recent literature. Nearly half of the articles (20 of 41) looked at the eva-

luation implementation category, and an equal number (20 of 41) examined the decision- or

policy-setting category. The characteristics under each of these categories were all examined

in at least one article, with the most prevalent characteristic, communication quality, of

Cousins and Leithwood appearing in 11.

However, as suggested by Shulha and Cousins (1997), changes in the conceptualizations

about use have occurred, so new characteristics might be expected to emerge. In fact, 25 of

the 41 studies in this review examined elements that were not covered by the 1986 framework.

Consequently, we added one characteristic—evaluator competence—to the evaluation

implementation category. In addition, we created an entirely new category—stakeholder

involvement—to accommodate the categorization of the 25 studies that examined aspects

of evaluation use that were not represented in the original Cousins and Leithwood framework.




Evaluator competence. This is a new characteristic under evaluation implementation that

has emerged since the development of the Cousins and Leithwood framework. Of the 41 stud-

ies in this review, six addressed the characteristics of evaluators, suggesting that evaluation

professionals play an important role in conducting evaluations that get used, albeit for different

reasons. Although the characteristic of credibility by Cousins and Leithwood gave some con-

sideration to the evaluator’s title or reputation, the definition did not extend to the influential

nature of the evaluator’s personal competence or leadership as a means of affecting the level of

evaluation use. Moreover, whereas the credibility characteristic addresses what the evaluator

does (e.g., methods selected, criteria used), the new evaluator competence characteristic

focuses more on who the evaluator is.

Stakeholder involvement. This is a new category that has been added to the original Cousins

and Leithwood framework to account for more recent research. The addition of this category

reflects the increased research focused on participatory evaluation approaches, stakeholder or

decision-maker participation, and/or stakeholder or decision-maker involvement since 1985.

Under the rubric of stakeholder involvement, we have identified nine characteristics. Eight

of them mirror those identified by Cousins and Leithwood but with the addition of involve-

ment to each. The original framework included research on the impact of direct decision-

maker involvement on use under commitment or receptiveness to evaluation. However, in the

current review, over half (23 of 41) of the studies addressed involvement, and the bulk of these

suggested that it was related to other category characteristics in their relationship with use.

Using the resulting modified Cousins and Leithwood framework, we classified the 41 stud-

ies of evaluation use from 1986 to 2005 according to 3 categories and 22 specific character-

istics. The most frequently studied characteristics were ‘‘involvement and commitment/

receptiveness to evaluation’’ (14 studies), followed by communication quality (11 studies) and

personal characteristics of users (9 studies). The least frequently studied characteristics were

‘‘involvement and information needs’’ and ‘‘involvement and decision characteristics,’’ each

appearing in a single study. About 40% of the studies (16 of 41) examined only a single char-

acteristic, with half of that group (8 of 16) studying a characteristic under the stakeholder

involvement category. The remainder of the studies examined multiple characteristics, ranging

from two to nine characteristics per study.

Table 1 defines each category, presents its related characteristics, and lists the studies that

examined each. Because the variables described in the studies did not always allow for obvious

categorization into the framework, this represents the authors’ best effort at accurately inter-

preting and deciding where the studies fit. The Appendix provides a summary of each study’s

focus, types of use, sample, categories, and findings. In terms of the types of evaluation use,

the information presented in the Appendix shows that the clear majority of the studies focused

on use of findings rather than process use. Only three studies examined process use, perhaps

because the concept of explicit process use is fairly recent. Within the use of findings, instru-

mental use was studied more frequently than conceptual use, which was typically linked to

instrumental use when researchers asked respondents whether actions were likely to be taken.

There were only a few studies that examined symbolic use.

Discussion

The purpose of this study was to review empirical research on evaluation use for the 25-year

period between 1986 and 2005. Basing the review on the framework of Cousins and

Leithwood allowed a comparison over time, and including other types of research (e.g.,




Ta

ble

1

Stu

die

sE

xa

min

ing

Use

by

Ca

teg

ory

an

dC

ha

ract

eris

tics

of

Va

ria

ble

s

Cat

egory

Char

acte

rist

icD

escr

ipti

on

of

Char

acte

rist

ic#

of

Stu

die

sR

elat

ionsh

ipto

Eval

uat

ion

Use

Art

icle

sth

atS

tudie

dC

har

acte

rist

ic

Eval

uat

ion

imple

men

tati

on

Com

munic

atio

n

qu

alit

y

Cla

rity

and

freq

uen

cyof

report

ing

resu

lts,

eval

uat

or

advoca

cyfo

r

resu

lts,

bre

adth

of

dis

sem

inat

ion.

Als

oin

cludes

the

type

of

reco

m-

men

dat

ions

inth

ere

port

and

the

pro

cess

of

com

munic

atio

n

bet

wee

nev

aluat

ors

and

clie

nts

11

Fre

quen

tly

among

the

most

import

ant

elem

ents

rela

ted

to

eval

uat

ion

use

.D

etai

led,

acti

onab

le,

evid

ence

-bas

ed

reco

mm

endat

ions

incr

ease

d

use

.B

yco

ntr

ast,

two

studie

s

found

no

rela

tionsh

ipw

ith

use

Bober

and

Bar

tlet

t(2

004)

Boyer

and

Lan

gbei

n(1

991)

Chin

(2003)

Eis

endra

th(1

988)

Johnst

on

(1986)

Mal

en,

Murp

hy,

and

Gea

ry(1

988)

Mar

ra(2

003)

Mar

shan

dG

lass

ick

(1988)

Rock

wel

l,D

ickey

,an

dJa

sa(1

990)

Shea

(1991)

Sle

ezer

(1987)

Tim

elin

ess

Tim

ing

of

the

eval

uat

ion

inla

rger

conte

xt;

tim

elin

ess

of

report

ing

when

eval

uat

ion

isco

mple

ted;

tim

ing

of

dis

sem

inat

ion

todec

isio

n

mak

ers

7M

ost

found

posi

tive

rela

tionsh

ip

bet

wee

nti

min

gan

dev

aluat

ion

use

.O

ne

study

found

that

tim

elin

ess

was

not

import

ant

in

det

erm

inin

guse

Bam

ber

ger

(2004)

Bar

rios

(1986)

Bober

and

Bar

tlet

t(2

004)

Boyer

and

Lan

gbei

n(1

991)

Eis

endra

th(1

988)

Rock

wel

let

al.

(1990)

Shea

(1991)

Eval

uat

or

com

pet

ence

a

Per

sonal

char

acte

rist

ics

of

the

eva-

luat

or

outs

ide

the

eval

uat

ion

pro

-

cess

,le

vel

of

cult

ura

lco

mpet

ence

,

lead

ersh

ipst

yle

of

eval

uat

or

6M

ost

studie

ssu

gges

tth

atev

alua-

tor

com

pet

ence

isim

port

ant

to

eval

uat

ion

use

Bar

rios

(1986)

Boyer

and

Lan

gbei

n(1

991)

Cal

lahan

,T

om

linso

n,

Hunsa

ker

,

Bla

nd,

and

Moon

(1995)

Cousi

ns

(1996)

Gre

ene

(1987)

Shea

(1991)

Eval

uat

ion

qual

ity

Char

acte

rist

ics

of

the

eval

uat

ion

pro

cess

,so

phis

tica

tion

of

met

hods,

rigor,

type

of

eval

uat

ion

model

6S

om

est

udie

sfo

und

ali

nk

bet

wee

nqual

ity

and

use

,

alth

ough

less

import

ant

than

reco

mm

endat

ions

and

com

mu-

nic

atio

n.

One

study

did

not

find

rela

tionsh

ipbet

wee

n

qual

ity

and

use

Bam

ber

ger

(2004)

Bober

and

Bar

tlet

t(2

004)

Johnst

on

(1986)

Rock

wel

let

al.

(1990)

Shea

(1991)

Pott

s(1

998)

(conti

nued

)


383



Ta

ble

1.

(co

nti

nu

ed)

Cat

egory

Char

acte

rist

icD

escr

ipti

on

of

Char

acte

rist

ic#

of

Stu

die

sR

elat

ionsh

ipto

Eval

uat

ion

Use

Art

icle

sth

atS

tudie

dC

har

acte

rist

ic

Fin

din

gs

Nat

ure

of

findin

gs

(e.g

.,posi

tive

or

neg

ativ

e),

exte

nt

of

congru

ence

wit

hau

die

nce

expec

tati

ons,

val

ue

of

findin

gs

for

dec

isio

nm

akin

g

6M

ixed

concl

usi

ons.

Intw

ost

ud-

ies

findin

gs

wer

eim

port

ant

to

use

,th

ough

less

soth

anco

m-

munic

atio

n,

tim

elin

ess,

and

eval

uat

ion

qual

ity

Bar

rios

(1986)

Bober

and

Bar

tlet

t(2

004)

Boyer

and

Lan

gbei

n(1

991)

Johnso

n(1

993)

Mal

enet

al.

(1988)

Wei

ss,

Murp

hy-G

raham

,an

d

Bir

kel

and

(2005)

Rel

evan

ceE

xte

nt

tow

hic

hth

ein

form

atio

npro

-

vid

edin

the

eval

uat

ion

isre

levan

t

toth

edec

isio

nm

aker

,an

dth

e

org

aniz

atio

nal

loca

tion

of

the

eval

uat

or

6M

ixed

concl

usi

ons.

Tw

ost

udie

s

did

not

find

rele

van

ceto

be

import

ant

touse

,but

two

studie

sfo

und

stro

nger

rela

-

tionsh

ips

bet

wee

nin

form

atio

n

rele

van

cean

duse

Bar

rios

(1986)

Bober

and

Bar

tlet

t(2

004)

Boyer

and

Lan

gbei

n(1

991)

Cousi

ns

(1995)

Gre

ene

(1987)

Shea

(1991)

Cre

dib

ilit

yT

he

obje

ctiv

ity,

bel

ievab

ilit

y,

and

appro

pri

aten

ess

of

the

eval

uat

ion

pro

cess

and/o

rof

the

acti

vit

ies

of

the

eval

uat

or

4S

pli

tfi

ndin

gs.

Tw

ost

udie

sfo

und

stro

ng

rela

tionsh

ipw

ith

eva-

luat

ion

use

;tw

ost

udie

sfo

und

no

such

rela

tionsh

ip

Bar

rios

(1986)

Bober

and

Bar

tlet

t(2

004)

Boyer

and

Lan

gbei

n(1

991)

Johnso

n(1

993)

Dec

isio

nor

poli

cy

sett

ing

Per

sonal

char

acte

rist

ics

Char

acte

rist

ics

of

the

eval

uat

ion

use

r,

for

exam

ple

,org

aniz

atio

nal

role

of

dec

isio

nm

aker

,in

form

atio

npro

-

cess

ing

style

,so

cial

char

acte

ris-

tics

,an

dso

on

9D

iffe

rence

sin

use

rs’

lear

nin

g

style

s,jo

bposi

tions,

adm

inis

-

trat

ive

level

,an

dex

per

ience

level

infl

uen

ceth

euse

of

eval

uat

ions

Bober

and

Bar

tlet

t(2

004)

Boyer

and

Lan

gbei

n(1

991)

Car

pin

ello

(1989)

Com

bs

(1999)

Cro

tti

(1993)

Ear

l(1

995)

Hopst

ock

,Y

oung,

and

Zeh

ler

(1993)

Mar

ra(2

003)

San

thiv

eera

n(1

995)

Com

mit

men

tan

d/

or

rece

pti

ven

ess

toev

aluat

ion

Use

rat

titu

des

tow

ard

the

eval

uat

ion

and

com

mit

men

tto

conduct

ing

eval

uat

ion;

the

exte

nt

tow

hic

hth

e

org

aniz

atio

nis

resi

stan

tto

eval

ua-

tion;

the

open

-min

ded

nes

sof

eva-

luat

ion

stak

ehold

ers

8S

om

est

udie

sfo

und

that

com

mit

-

men

t,ac

tive

org

aniz

ing

effo

rts,

and

support

ive

bac

ker

s

incr

ease

duse

.O

ne

study

found

that

atti

tude

tow

ard

eval

uat

ion

did

not

affe

ctuse

Boyer

and

Lan

gbei

n(1

991)

Cro

tti

(1993)

Johnso

n(1

993)

Mal

enet

al.

(1988)

Mar

ra(2

003)

McC

orm

ick

(1997)

Rin

ne

(1994)

San

thiv

eera

n(1

995)

(conti

nued

)


384



Poli

tica

lcl

imat

eT

he

poli

tica

lori

enta

tion

of

the

peo

ple

who

com

mis

sioned

the

eval

uat

ion,

the

exte

nt

tow

hic

hdec

isio

nm

aker

isdep

enden

ton

exte

rnal

sponso

rs,

inte

rnal

rival

ries

,budget

fights

,

and

pow

erst

ruggle

s

6G

ener

ally

,at

tendin

gto

poli

tica

l

clim

ate

was

found

toin

crea

se

use

Eis

endra

th(1

988)

Had

dock

(1998)

Johnst

on

(1986)

Mal

enet

al.

(1988)

San

thiv

eera

n(1

995)

Wei

sset

al.

(2005)

Dec

isio

n

char

acte

rist

ics

The

signif

ican

ceof

the

dec

isio

nor

eval

uat

ion

pro

ble

m,

the

type

of

dec

isio

nto

be

mad

e,th

enovel

tyof

the

pro

gra

mar

ea

5E

ach

of

the

five

studie

sre

port

ed

connec

tions

bet

wee

ndec

isio

n

char

acte

rist

ics

and

eval

uat

ion

use

Bar

rios

(1986)

Bro

wn-M

cGow

an(1

992)

Eis

endra

th(1

988)

Mal

enet

al.

(1988)

New

man

,B

row

n,

and

Riv

ers

(1987)

Com

pet

ing

info

rmat

ion

Info

rmat

ion

rela

ted

toth

esu

bje

ctof

the

eval

uat

ion

and

avai

lable

to

stak

ehold

ers

from

outs

ide

the

eva-

luat

ion

pro

cess

,th

atis

,th

rough

per

sonal

obse

rvat

ion,

that

com

-

pet

esw

ith

eval

uat

ion

dat

a

3C

ontr

adic

tory

findin

gs.

One

study

found

that

ala

rge

amount

of

com

pet

ing

info

rmat

ion

did

not

affe

ctin

stru

men

tal

use

,

wher

eas

anoth

erfo

und

that

hig

h-l

evel

poli

cyoff

icia

lsuse

d

the

eval

uat

ion

resu

lts

only

when

they

wer

esu

pport

edby

oth

erso

urc

esof

info

rmat

ion

Eis

endra

th(1

988)

Johnso

n(1

993)

Wei

sset

al.

(2005)

Info

rmat

ion

nee

ds

of

the

eval

ua-

tion

audie

nce

s

Info

rmat

ion

nee

ds

of

the

eval

uat

ion

audie

nce

,th

ety

pes

of

info

rmat

ion,

the

num

ber

of

audie

nce

sw

ith

dif

-

feri

ng

info

rmat

ion

nee

ds,

tim

e

pre

ssure

,an

dper

ceiv

ednee

dfo

r

eval

uat

ion

2B

oth

studie

sfo

und

that

atte

ndin

g

toth

eau

die

nce

’sin

form

atio

n

nee

ds

posi

tivel

yin

fluen

ced

the

use

of

eval

uat

ion

resu

lts

Hopst

ock

etal

.(1

993)

Rin

ne

(1994)

(conti

nued

)


385



Ta

ble

1.

(co

nti

nu

ed)

Cat

egory

Char

acte

rist

icD

escr

ipti

on

of

Char

acte

rist

ic#

of

Stu

die

sR

elat

ionsh

ipto

Eval

uat

ion

Use

Art

icle

sth

atS

tudie

dC

har

acte

rist

ic

Sta

keh

old

er

involv

emen

t

Involv

emen

tw

ith

com

mit

men

tor

rece

pti

ven

ess

to

eval

uat

ion

Involv

ing

eval

uat

ion

stak

ehold

ers

crea

tes

aco

mm

itm

ent

or

rece

p-

tiven

ess

toev

aluat

ion

14

For

the

most

par

t,co

mm

itm

ent

that

was

stre

ngth

ened

by

involv

emen

tin

the

eval

uat

ion

was

found

toposi

tivel

yin

flu-

ence

eval

uat

ion

use

.In

one

study,

the

involv

emen

tof

a

com

mit

ted

exec

uti

ve

off

icer

was

esse

nti

alto

the

imple

-

men

tati

on

of

eval

uat

ion

findin

gs

Alt

schuld

,Y

oon,

and

Cull

en(1

993)

Ayer

s(1

987)

Bar

rios

(1986)

Bro

wn-M

cGow

an(1

992)

Cal

lahan

etal

.(1

995)

Ear

l(1

995)

Eis

endra

th(1

988)

Gre

ene

(1987)

Gre

ene

(1988)

Had

dock

(1998)

Laf

leur

(1995)

Lee

and

Cousi

ns

(1995)

Rock

wel

let

al.

(1990)

Shea

(1991)

Involv

emen

tw

ith

com

munic

atio

n

qu

alit

y

Sta

keh

old

erin

volv

emen

tpro

mote

s

impro

ved

com

munic

atio

n

5A

llfi

ve

studie

sid

enti

fied

way

sin

whic

hst

akeh

old

erin

volv

emen

t

led

togre

ater

use

Bam

ber

ger

(2004)

Cousi

ns

(1995)

Fors

s,C

rack

nel

l,an

dS

amse

t(1

994)

Gre

ene

(1988)

Laf

leur

(1995)

Dir

ect

stak

ehold

er

involv

emen

t

The

dir

ect

rela

tionsh

ipbet

wee

n

involv

emen

tan

dev

aluat

ion

use

4A

llst

udie

sre

port

edin

volv

e-

men

t’s

posi

tive

infl

uen

ceon

var

ious

types

of

use

Cai

(1996)

Pre

skil

lan

dC

arac

elli

(1997)

Sper

lazz

a(1

995)

Turn

bull

(1999)

Involv

emen

tw

ith

cred

ibil

ity

Sta

keh

old

erin

volv

emen

tle

dto

incr

ease

dcr

edib

ilit

yof

the

eva-

luat

ion

pro

cess

and/o

rth

e

eval

uat

or

4T

hre

eof

the

four

studie

sobse

rved

ast

rong

rela

tionsh

ipw

ith

use

Cousi

ns

(1995)

Gre

ene

(1987)

Laf

leur

(1995)

Shea

(1991)

Involv

emen

tw

ith

findin

gs

Involv

ing

eval

uat

ion

stak

ehold

ers

in

know

ing

and

under

stan

din

gth

e

eval

uat

ion

findin

gs

4T

hre

est

udie

sem

phas

ized

that

involv

emen

tre

late

dto

the

findin

gs

was

import

ant

toev

a-

luat

ion

use

Cousi

ns

(1995)

Gre

ene

(1987)

Laf

leur

(1995)

Shea

(1991)

(conti

nued

)


386



Involv

emen

tw

ith

rele

van

ce

Sta

keh

old

erpar

tici

pat

ion

toin

tegra

te

import

ant

org

aniz

atio

nal

conce

rns

into

the

eval

uat

ion

des

ign

4F

or

the

most

par

t,in

crea

sed

con-

tact

wit

hst

akeh

old

ers

fost

ered

incr

ease

dre

levan

ceth

at

resu

lted

inin

crea

sed

eval

ua-

tion

use

Cousi

ns

(1995)

Gre

ene

(1987)

Laf

leur

(1995)

Shea

(1991)

Involv

emen

tw

ith

per

sonal

char

acte

rist

ics

Involv

emen

tof

eval

uat

ion

stak

e-

hold

ers

atdif

fere

nt

org

aniz

atio

nal

level

s

2F

indin

gs

of

one

study

sugges

tth

at

involv

emen

tof

man

ager

s

affe

cts

use

more

exte

nsi

vel

y

than

involv

ing

oth

erst

aff

Cousi

ns

(1995)

McC

orm

ick

(1997)

Involv

emen

tw

ith

dec

isio

n

char

acte

rist

ics

Involv

ing

of

ara

nge

of

stak

ehold

ers

indif

fere

nt

sett

ings

dep

endin

gon

the

char

acte

rist

ics

of

the

dec

isio

n

that

nee

ds

tobe

mad

e

1T

his

study

found

aposi

tive

rela

-

tionsh

ipbet

wee

nev

aluat

ion

use

and

involv

emen

tby

indi-

vid

ual

sin

nontr

adit

ional

bure

aucr

acie

sw

her

edec

isio

n

mak

ing

involv

esin

put

from

peo

ple

atal

lle

vel

sin

the

org

aniz

atio

n

Johnso

n(1

993)

Involv

emen

tw

ith

info

rmat

ion

nee

ds

The

involv

emen

tof

stak

ehold

ers

faci

lita

ted

the

intr

oduct

ion

of

thei

r

info

rmat

ion

nee

ds

1In

volv

edst

akeh

old

ers’

des

ire

for

info

rmat

ion

and

the

tim

elin

ess

of

the

eval

uat

ion

fost

ered

info

rmat

ion

ow

ner

ship

,w

hic

h

was

posi

tivel

yre

late

dto

use

Rock

wel

let

al.

(1990)

aE

val

uat

or

com

pet

ence

was

not

aca

tegory

inth

eC

ousi

ns

and

Lei

thw

ood

fram

ework

,but

auth

ors

pro

pose

itas

anew

char

acte

rist

icin

the

eval

uat

ion

imple

men

tati

on

cate

gory

.


387



dissertations) broadened its scope. This literature review located 41 empirical studies of

evaluation use conducted between 1986 and 2005 that met minimum quality standards.

Most of the studies (38 of 41) examined the use of findings rather than process use; only

three studies examined process use. The lack of attention to process use in the articles included

in this review might have resulted from the fact that the concept of explicit process use is fairly

recent, and the field is still more focused on outcomes and results. Alternatively, it might be

that empirical studies are more likely to focus on the use of results because measuring process

use is less well defined. Finally, the limited attention to process use might have resulted from

our search strategy, which excluded evaluation capacity building studies, many of which mea-

sured organizational learning through the evaluation process. These studies are not included in

this review but are synthesized in a publication by Cousins et al. (2004). After the findings

were categorized according to the Cousins and Leithwood framework, one additional category

(stakeholder involvement) and one new characteristic (evaluator competence) emerged. These

additions align with the comments of Shulha and Cousins (1997) made more than 10 years ago

about changes in the field, especially the diversification of the evaluator’s role.

The stakeholder involvement category reflects the expansion of participatory evaluation

methods. The framework of Cousins and Leithwood included stakeholder involvement under

the ‘‘commitment and/or receptiveness to evaluation’’ characteristic within the decision- and

policy-setting category. This was sufficient in the mid-1980s because only 10% of the studies

in their review included involvement, and these were all related to the effects of involvement

on stakeholders’ commitment or receptiveness to evaluation. In addition, four of the studies in

the current review directly examined the relationship between stakeholder involvement and

evaluation. This dynamic was not present in any of the studies examined by Cousins and Leith-

wood. The emergence of this new category suggests that evaluators may want to focus on

involving stakeholders as a way to enhance evaluation use. The addition of the evaluator com-

petence characteristic indicates a growing acknowledgment of the importance of the compe-

tence of individual evaluators, both professionally and culturally—and the value of these

characteristics in efforts to increase evaluation use.

Some studies—Shea (1991), Bober and Bartlett (2004), Boyer and Langbein (1991), and

Malen, Murphy, and Geary (1988)—examined multiple characteristics. It seemed possible that

these studies might help us think about evaluation influence by identifying important variables

in a sequence suggestive of a pathway, at least at the individual level. This effort failed because

the studies examined variables related to use, not pathways leading to it. Identifying pathways

was a creative activity rather than a way to summarize the research. As Weiss et al. (2005)

found when they sought influence pathways after the fact in their drug abuse resistance edu-

cation (DARE) study, ‘‘We became bogged down in unique tangles of strings [of pathways]

. . . . We are on less sure ground trying to reconstruct individual and interpersonal processes

that were reported to us some 2 to 8 years after the events.’’ In other words, the existing empiri-

cal research on evaluation use has identified a collection of important variables, but research

on influence pathways will necessitate a different strategy. In settings that have specific out-

come variables and sufficient interval data on other variables, path analysis might be one

potential method. Future research might focus on developing quantitative outcome and process

measures that could then be used to gather enough data to conduct path analyses and determine

models displaying the relationships among the process measures and the outcomes.

It is impossible, finally, to answer the question of which characteristics are most related to

increasing the use of evaluations in a straightforward manner. A meta-analysis of the studies is not

possible because the studies do not operationalize or measure the variables in the same manner.

Cousins and Leithwood compensated for this problem by creating a quantitative index that

weighed the number of positive, negative, and nonsignificant findings for each characteristic to

create a ‘‘prevalence of relationship’’ index. Based on this index, they concluded that evaluation




quality and decision characteristics were most highly related to use, followed by evaluation find-

ings, users’ commitment or receptiveness to evaluation, and evaluation relevance.

This index provides a means of comparing findings across a variety of studies. Unfortu-

nately, drawing conclusions about which characteristics are related to use remains problematic

because this type of meta-synthesis is highly affected by the components that researchers

chose to include, and it may not include what is actually occurring. In addition, the publication

process may exclude studies with inconclusive or negative findings. Instead, the current study

discusses those elements that appear to be most ‘‘empirically supported’’—meaning those ele-

ments that are both highly studied and supported by strong evidence of a positive relationship

to evaluation use. Reframing the conversation to discuss ‘‘empirically supported’’ character-

istics also allows the suggestion of evidence-based practices that evaluators can employ to

increase the use of their evaluations.

Framed with these cautions in mind, we identified the following empirically supported fac-

tors that promote the use of evaluation. Findings highlight the importance of stakeholder invol-

vement in facilitating evaluation use. In several studies, involvement was found to facilitate an

evaluation process that, in turn, improved the evaluation implementation characteristics. In

other studies, stakeholder involvement supported decision making or policy setting that fos-

tered greater capacity for using evaluation information. Stated differently, stakeholder invol-

vement is a mechanism that facilitates those aspects of an evaluation’s process or setting that

lead to greater use. More than just involvement by stakeholders or decision makers alone, how-

ever, the findings from this literature review suggest that engagement, interaction, and com-

munication between evaluation clients and evaluators is key to maximizing the use of the

evaluation in the long run.

Limitations

Features of the research method used in this study, particularly the choice to limit the

review to empirical studies of evaluation use conducted between 1986 and 2005, precluded

consideration of any theoretical articles on evaluation produced during that time period. This

fact is not intended to detract from the positive contributions to the understanding of evalua-

tion use made by the authors of these articles. In addition, the research design included a deci-

sion to limit the search terms to ‘‘evaluation utilization,’’ ‘‘evaluation use,’’ and ‘‘evaluation

influence.’’ This decision resulted in the exclusion of ‘‘evaluation capacity building’’ studies

that examined organizational learning through the evaluation process—one form of use—but

did not include the keywords ‘‘use’’ or ‘‘utilization.’’ Finally, the sample sizes of some of the

studies included in this review are rather small. Of the 41 studies included in the review,

approximately 65% (19 of 41) have sample sizes of 12 or fewer. The remaining studies ranged

in sample size from 26 to 540.

Conclusion

In summary, the findings from this literature review support Cousins’ (2003) conceptual

framework that outlines dimensions of ‘‘evaluation context’’ (similar to evaluation implemen-

tation characteristics) and ‘‘decision/policy setting.’’ Additionally, the findings support the addi-

tion of one new category—stakeholder involvement—and one new characteristic—evaluator

competence (under the category of evaluation implementation). Findings point to the importance

of stakeholder involvement in facilitating evaluation use and suggest that engagement, interac-

tion, and communication between evaluation clients and evaluators is critical to the meaningful

use of evaluations.




Ap

pen

dix

Su

mm

ary

of

Em

pir

ica

lS

tud

ies

of

Ev

alu

ati

on

Use

an

dIn

flu

ence

(19

86

–2

00

5)

Stu

dy

Type

of

Use

Focu

sof

Stu

dy

Sam

ple

Cat

egory

of

Use

Key

Fin

din

gs

Fin

din

gs

Use

ProcessUse

Instrument

Conceptual

Symbolic

Alt

schuld

etal

.

(1993)

pp

Rel

atio

nsh

ipbet

wee

nat

titu

des

tow

ard

nee

ds

asse

ssm

ent,

invol-

vem

ent

inpro

cess

,bac

kgro

und

char

acte

rist

ics,

and

report

ing

char

acte

rist

ics

and

the

conce

ptu

al

and

inst

rum

enta

luti

liza

tion

of

nee

ds

asse

ssm

ent

concl

usi

ons

Hig

her

educa

tion

adm

inis

trat

ors

(n¼

62)

Dec

isio

nor

poli

cyse

ttin

g;

sta-

keh

old

erin

volv

emen

t

Use

of

nee

ds

asse

ssm

ents

wer

e

infl

uen

ced

by

coll

ege

adm

inis

-

trat

ors

’at

titu

des

and

level

sof

involv

emen

t.T

he

adm

inis

trat

ors

’

bac

kgro

und/t

rain

ing

and

char

ac-

teri

stic

sof

the

nee

ds

asse

ssm

ent

report

sw

ere

not

found

tobe

rela

ted

touse

Ayer

s(1

987)

pp

Rel

atio

nsh

ipbet

wee

nuse

of

a

‘‘st

akeh

old

erco

llab

ora

tive’

’ev

a-

luat

ion

appro

ach

and

inst

rum

en-

tal

and

conce

ptu

aluse

Guam

publi

csc

hool

dis

tric

t(n¼

1)

Dec

isio

nor

poli

cyse

ttin

g;

sta-

keh

old

erin

volv

emen

t

Ayer

sin

terv

iew

edfo

ur

of

the

sta-

keh

old

ers

who

par

tici

pat

edin

all

phas

esof

the

eval

uat

ion,

asw

ell

astw

om

ajor

use

rsof

the

eva-

luat

ion,

toso

lici

tper

cepti

ons

of

the

pro

cess

and

of

subse

quen

t

use

.P

arti

cipan

tsre

port

edposi

-

tive

atti

tudes

tow

ard

the

pro

cess

,

but

dir

ect

use

of

the

report

was

low

.H

ow

ever

,al

though

use

,as

mea

sure

dby

imple

men

tati

on

of

reco

mm

endat

ions,

was

low

,th

e

findin

gs

trig

ger

edpla

nnin

gdis

-

cuss

ions

and

neg

oti

atio

ns

bet

wee

nunio

nan

dag

ency

adm

inis

trat

ion

(conti

nued

)


390



Bam

ber

ger

(2004)

pp

Char

acte

rist

ics

of

hig

hly

cost

-

effe

ctiv

eev

aluat

ions

of

inte

rna-

tional

dev

elopm

ent

pro

ject

s

Dev

elopm

ent

pro

ject

eval

uat

ions

(n¼

8)

Eval

uat

ion

imple

men

tati

on;

dec

isio

nor

poli

cyse

ttin

g;

stak

ehold

erin

volv

emen

t

Iden

tifi

edfi

ve

fact

ors

that

incr

ease

d

the

impac

tof

anev

aluat

ion:

(a)

a

conduci

ve

poli

cyen

vir

onm

ent—

eval

uat

ion

addre

sses

curr

ent

conce

rns

and

ther

eis

aco

mm

it-

men

tby

dec

isio

nm

aker

sto

use

resu

lts;

(b)

tim

ing

of

eval

uat

ion—

eval

uat

ion

launch

edw

hen

ther

e

are

clea

rly

def

ined

info

rmat

ion

nee

ds;

(c)

role

of

eval

uat

ion—

eval

uat

or

mu

stun

ders

tand

eva-

luat

ion

ison

eso

urc

eof

data

wit

hin

adeci

sion

-mak

ing

con-

text;

(d)

bu

ildin

ga

rela

tion

ship

wit

hth

ecl

ient

and

effe

ctiv

ely

com

mun

icat

ing

find

ings;

and

(e)

eval

uat

ion

cond

ucte

dby

eith

er

the

evalu

atio

nun

itof

the

man-

agin

gor

fund

ing

agen

cyor

by

ou

tsid

eag

ency,

or

join

tly,

asth

e

conte

xt

dic

tate

s

Bar

rios

(1986)

ppp

Rel

atio

nsh

ipbet

wee

nte

chnic

alan

d

org

aniz

atio

nal

var

iable

san

d

inst

rum

enta

l,co

nce

ptu

al,

and

per

suas

ive

use

of

eval

uat

ion

info

rmat

ion

Sta

te-l

evel

soci

alse

rvic

eag

ency

(n¼

1)

Eval

uat

ion

imple

men

tati

on;

dec

isio

nor

poli

cyse

ttin

g;

stak

ehold

erin

volv

emen

t

Rec

om

men

dat

ions

requir

ing

poli

cy

chan

ges

or

inte

rpro

gra

mor

inte

ragen

cyac

tion

wer

em

ore

infl

uen

tial

inte

rms

of

the

dec

i-

sions

toim

ple

men

tth

emin

com

par

ison

wit

hre

com

men

da-

tions

that

sugges

ted

only

acti

on

by

pro

gra

mm

anag

ers.

The

fol-

low

ing

var

iable

sar

eal

sore

late

d

touti

liza

tion:use

rin

volv

emen

tin

the

form

ula

tion

of

the

study

and

eval

uat

or

cred

ibil

ity

inte

rms

of

pro

gra

mknow

ledge

(conti

nued

)


391



Bober

and

Bar

tlet

t

(2004)

pp

Eval

uat

ion

imple

men

tati

on

fact

ors

and

dec

isio

nan

dpoli

cyse

ttin

g

fact

ors

affe

ctin

gth

euse

of

trai

n-

ing

eval

uat

ion

resu

lts

atco

rpora

te

univ

ersi

ties

Corp

ora

teuniv

ersi

ties

(n¼

4)

Eval

uat

ion

imple

men

tati

on;

dec

isio

nor

poli

cyse

ttin

g

Corp

ora

teuniv

ersi

tym

anag

ers

use

d

eval

uat

ion

findin

gs

ina

var

iety

of

way

sw

ith

inst

rum

enta

luse

s

dom

inat

ing.

Eval

uat

ion

imple

-

men

tati

on

fact

ors

wer

em

ore

import

ant

than

dec

isio

n-

or

poli

cy-s

etti

ng

fact

ors

inim

pac

t-

ing

use

.T

he

most

hig

hly

ranked

fact

or

was

com

munic

atio

nqual

-

ity.

Use

of

mult

iple

met

hods

of

report

ing

dat

aw

asef

fect

ive

for

incr

easi

ng

use

Boyer

and

Lan

gbei

n

(1991)

pF

acto

rsre

late

dto

the

use

of

hea

lth-

rela

ted

eval

uat

ion

rese

arch

resu

lts

by

mem

ber

sof

congre

ss

and

congre

ssio

nal

staf

fers

Congre

ssio

nal

hea

lth

and

hea

lth-

rela

ted

staf

fm

ember

s(n¼

100)

Eval

uat

ion

imple

men

tati

on;

dec

isio

nor

poli

cyse

ttin

g;

eval

uat

or

com

pet

ence

Congre

ssio

nal

mem

ber

san

dst

affe

rs

bel

ieved

eval

uat

ion

report

sto

be

rele

van

t,ti

mel

y,

clea

r,m

ethodo-

logic

ally

rigoro

us,

and

pro

duce

d

by

reputa

ble

pra

ctit

ioner

s.T

he

rela

tive

import

ance

of

fact

ors

affe

ctin

guse

var

ied

dep

endin

gon

what

type

of

report

(Gen

eral

Acc

ounti

ng

Off

ice

[GA

O]

vs.

non-G

AO

)an

duse

r(m

ember

of

congre

ssvs.

staf

fer)

.O

ver

all,

tim

elin

ess

of

GA

Ore

port

sw

asth

e

stro

nges

tfa

ctor,

wit

hcr

edib

ilit

y

of

met

hodolo

gy,

and

clar

ity

of

report

ing

also

bei

ng

import

ant.

Pre

sence

of

anad

voca

teor

abse

nce

of

adet

ract

or

of

the

eva-

luat

or

also

pla

yed

aro

lein

use

Bro

wn-

McG

ow

an

(1992)

pE

ffec

tof

know

ledge

use

syst

em

(KU

S)

on

use

of

eval

uat

ion

find-

ings;

rela

tionsh

ipbet

wee

nev

a-

luat

ion

pro

cess

,si

gnif

ican

ceof

the

dec

isio

n,

per

ceiv

edim

pac

ts

of

the

dec

isio

n,

and

pre

fere

nce

s

tow

ard

eval

uat

ion

outc

om

esan

d

uti

liza

tion

of

eval

uat

ion

resu

lts

Sen

ior

hig

her

educa

tion

adm

inis

-

trat

ors

(n¼

8)

Dec

isio

nor

poli

cyse

ttin

g;

stak

ehold

erin

volv

emen

t

Dec

isio

nm

aker

sre

port

edso

me

incr

ease

inth

eir

par

tici

pat

ion

and

inte

rest

inth

eev

aluat

ion

pro

cess

bec

ause

of

usi

ng

the

KU

S.

The

uti

liza

tion

of

eval

uat

ion

findin

gs

was

impro

ved

.T

he

eval

uat

ion

qual

ity

and

uti

liza

tion

of

resu

lts

wer

eal

soen

han

ced

by

dec

isio

n

mak

ers’

per

sonal

stak

esin

the

eval

uat

ion

(conti

nued

)


392



Cai

(1996)

ppp

Rel

atio

nsh

ipbet

wee

nte

acher

s’

per

cepti

ons

of

thei

rin

volv

emen

t

inpro

gra

mev

aluat

ion

and

report

edle

vel

sof

inst

rum

enta

l,

conce

ptu

al,

and

sym

boli

cuse

New

York

stat

eK

-12

publi

csc

hool

teac

her

s(n¼

207)

Sta

keh

old

erin

volv

emen

tC

urr

ent

opport

unit

yfo

rin

volv

e-

men

tis

rela

ted

tow

illi

ngnes

sto

par

tici

pat

ein

futu

reim

ple

men

ta-

tion.

Lev

elan

dphas

eof

invol-

vem

ent

inev

aluat

ion

isre

late

dto

per

ceiv

edben

efit

sto

indiv

idual

and

toorg

aniz

atio

n.

The

ben

efit

s

of

such

involv

emen

tin

clude:

enhan

ced

uti

liza

tion

and

wil

ling-

nes

sto

be

involv

edin

futu

re

eval

uat

ions,

incr

ease

dknow

ledge

and

skil

lsre

late

dto

eval

uat

ion,

and

impro

ved

com

munic

atio

n

pro

cess

wit

hin

org

aniz

atio

ns

Cal

lahan

etal

.

(1995)

pF

acto

rsan

dpra

ctic

esre

late

dto

eval

uat

ion

uti

liza

tion

ingif

ted

educa

tion

pro

gra

ms;

exam

inat

ion

of

exem

pla

ryan

dnonex

empla

ry

eval

uat

ions

and

exte

nt

of

imple

-

men

tati

on

of

eval

uat

ion

reco

mm

endat

ions

Eval

uat

ion

report

sfr

om

dis

tric

t

gif

ted

educa

tion

pro

gra

ms

(n¼

12)

Dec

isio

nor

poli

cyse

ttin

g;

eval

uat

or

com

pet

ence

;

stak

ehold

erin

volv

emen

t

All

12

dis

tric

tsuse

dev

aluat

ion

info

rmat

ion

toen

act

som

ech

ange

ingif

ted

educa

tion

pro

gra

mm

ing.

The

‘‘w

ill

and

skil

l’’

of

key

per

-

sonnel

toev

aluat

eaf

fect

edth

euse

of

the

eval

uat

ion

resu

lts.

Key

condit

ions

affe

ctin

guse

:(a

)

dis

tric

t-w

ide

eval

uat

ion

poli

cy;(

b)

wri

tten

pla

ns

on

how

toim

ple

men

t

findin

gs;

(c)

mult

iple

stak

ehold

ers

wer

eco

nsi

sten

tly

involv

edin

pla

nnin

g,m

onit

ori

ng,an

dre

vie

w-

ing

eval

uat

ion

pro

cess

and

find-

ings;

(d)

stak

ehold

ers

pla

yed

role

of

advoca

ting

for

pro

gra

mch

ange

bas

edon

findin

gs;

and

(e)

key

per

sonnel

wer

eaw

are

of

rela

tion-

ship

bet

wee

ngif

ted

ed,e

val

uat

ion,

and

poli

tica

lpro

cess

es

(conti

nued

)


393



Car

pin

ello

(1989)

pp

Exam

ines

the

effe

ctof

the

pow

er

bas

eof

the

eval

uat

or

(leg

itim

ate,

refe

rent,

or

exper

t),

per

cepti

ons

of

dec

isio

n-m

akin

gco

nse

-

quen

ces,

and

eval

uat

ion

use

r

exper

ience

son

eval

uat

ion

use

in

term

sof

agre

emen

tw

ith

reco

m-

men

dat

ions,

per

cepti

ons

of

eva-

luat

ion

cred

ibil

ity,

nee

ds

for

info

rmat

ion,

and

inst

rum

enta

l

dec

isio

ns

Ger

onto

logy

nurs

esfr

om

New

York

(n¼

282)

Dec

isio

nor

poli

cyse

ttin

gC

onse

quen

ce,

pow

er,

and

exper

i-

ence

wer

efo

und

toaf

fect

how

eval

uat

ion

info

rmat

ion

isuse

d

and

pro

cess

edby

nurs

edec

isio

n

mak

ers.

Exper

ience

ddec

isio

n

mak

ers

indic

ated

anee

dfo

r

info

rmat

ion

when

infl

uen

ced

by

econom

icco

nse

quen

ces

and

refe

rent

pow

erbas

es,

wher

eas

less

exper

ience

ddec

isio

nm

aker

s

wer

eaf

fect

edby

affe

ctiv

eco

n-

sequen

ces

and

exper

tpow

er

bas

es

Chin

(2003)

pp

Impac

tof

usi

ng

cart

oons

and

poet

ry

inev

aluat

ion

report

son

pro

mpt

dis

cuss

ion

of

findin

gs

and

incr

ease

dunder

stan

din

gof

resu

lts

by

eval

uat

ion

stak

ehold

ers

Sch

ool

dis

tric

tev

aluat

ion

stak

e-

hold

ers

(n¼

26)

Eval

uat

ion

imple

men

tati

on

Alt

hough

cart

oons

and

poet

ryw

ere

wel

lre

ceiv

edby

eval

uat

ion

sta-

keh

old

ers,

eval

uat

ors

wer

enot

as

support

ive

of

thei

rin

clusi

on.

The

poet

ryan

dca

rtoons

convey

edan

emoti

onal

and/o

rvis

ual

repre

-

senta

tion

of

findin

gs;

how

ever

,

this

did

not

incr

ease

dis

cuss

ion

of

the

findin

gs

among

stak

ehold

ers

nor

did

iten

sure

that

report

read

ers

clea

rly

per

ceiv

edth

e

auth

or’

sin

tended

mes

sages

Com

bs

(1999)

ppp

Rel

atio

nsh

ipbet

wee

npre

exis

ting

posi

tive

atti

tudes

tow

ard

incl

u-

sive

educa

tion

and

the

per

sua-

siven

ess

of

pro

gra

mev

aluat

ion

findin

gs

asm

easu

red

by

Russ

on

and

Koeh

ly(1

995)

per

suas

ion

scal

e

Gen

eral

and

spec

ial

educa

tion

ele-

men

tary

teac

her

sin

Nort

hC

aro-

lin

a(n¼

76)

Dec

isio

nor

poli

cyse

ttin

gA

lthough

the

study

found

that

teac

her

s’at

titu

des

tow

ard

incl

u-

sion

wer

epre

dic

tive

of

the

per

-

suas

iven

ess

of

the

sum

mar

y

eval

uat

ion

report

,co

ncl

usi

ons

from

the

study

are

lim

ited

by

the

pec

uli

arit

ies

of

the

dat

a

(untr

eate

doutl

iers

inth

edat

ase

t)

(conti

nued

)


394



Cousi

ns

(1995)

pp

Exam

inat

ion

of

the

impac

tof

par

ti-

cipat

ory

appro

aches

use

din

one

mar

gin

ally

succ

essf

ul

and

one

hig

hly

succ

essf

ul

educa

tional

eval

uat

ion

Can

adia

ned

uca

tion

fiel

dce

nte

rs

(n¼

2)

Eval

uat

ion

imple

men

tati

on;

stak

ehold

erin

volv

emen

t

The

par

tici

pat

ory

pro

cess

enhan

ced

cred

ibil

ity

of

the

report

and

mad

e

the

findin

gs

more

rele

van

t,w

hic

h

intu

rnin

crea

sed

the

report

ed

use

fuln

ess

of

the

eval

uat

ion

Cousi

ns

(1996)

pE

ffec

tsof

rese

arch

erin

volv

emen

t

level

son

exte

nt

and

type

of

rec-

om

men

dat

ion

imple

men

tati

on

Can

adia

nsc

hool

dis

tric

ts(n¼

3)

Eval

uat

or

com

pet

ence

Des

pit

eth

evar

yin

gle

vel

sof

rese

arch

erin

volv

emen

t,docu

-

men

ted

use

was

rela

tivel

yst

able

.

Use

appea

red

tobe

more

affe

cted

by

tim

epre

ssure

san

dad

min

is-

trat

ive

support

than

by

level

of

rese

arch

erin

volv

emen

t.In

the

low

est

involv

emen

tca

se,

pote

n-

tial

for

use

was

hig

her

than

actu

al

use

,giv

enth

eti

mef

ram

eof

the

eval

uat

ion

Cro

tti

(1993)ppp

Use

of

pro

cess

and

end

pro

duct

sof

Pen

nsy

lvan

ia’s

long-r

ange

pla

ns,

asper

ceiv

edby

school

adm

inis

-

trat

ors

;re

lati

onsh

ipbet

wee

n

hum

anan

dco

nte

xt

char

acte

ris-

tics

and

per

cepti

ons

of

use

fuln

ess

Pen

nsy

lvan

iasc

hool

dis

tric

ts

(n¼

11)

Dec

isio

nor

poli

cyse

ttin

gD

iffe

rent

adm

inis

trat

ive

level

s

emphas

ized

dif

fere

nt

form

sof

eval

uat

ion

use

.L

oca

lco

nst

rain

ts

had

min

imal

infl

uen

ceon

the

eval

uat

ion

uti

liza

tion

pro

cess

.

The

acti

ve

org

aniz

ing

effo

rts

of

school

adm

inis

trat

ors

report

edly

pro

mote

dlo

ng-r

ange

pla

nuti

li-

zati

on.

Fac

tor

clust

ers

com

pri

s-

ing

hum

anan

dev

aluat

ion

var

iable

sre

ceiv

edhig

her

over

all

import

ance

than

conte

xt

var

iable

s

(conti

nued

)


395



Ear

l(1

995)

pp

Exam

inat

ion

of

the

impac

tof

two

par

tici

pat

ory

eval

uat

ions

on

the

incr

ease

dunder

stan

din

g,

com

-

mit

men

t,an

duti

liza

tion

of

the

eval

uat

ion

by

the

eval

uat

or

and

clie

nts

Sch

ools

ina

Can

adia

nsc

hool

dis

-

tric

t(n¼

93)

Dec

isio

nor

poli

cyse

ttin

g;

stak

ehold

erin

volv

emen

t

Tw

opar

tici

pat

ory

eval

uat

ions

focu

sed

on

school

impro

vem

ent

ina

larg

esu

burb

ansc

hool

dis

-

tric

t.P

arti

cipan

tsco

uld

hav

e

bee

nin

volv

edon

thre

ele

vel

s.

The

leas

tin

volv

edte

ams

wer

e

inte

rvie

wed

for

the

eval

uat

ion.

The

moder

atel

yin

volv

edte

ams

had

mem

ber

sw

ho

serv

edas

inte

rvie

wer

s.T

he

most

involv

ed

team

spla

nned

the

inte

rvie

wpro

-

cess

and

pro

toco

ls.

Tea

ms

that

wer

eth

ele

ast

involv

ed(i

nte

r-

vie

wee

s)w

ere

slig

htl

yle

ssli

kel

y

than

the

moder

atel

yor

most

involv

edte

amm

ember

sto

report

posi

tive

feel

ings

about

the

pro

-

cess

.O

ver

all,

hig

huse

and

pote

nti

alfo

ruse

was

found

for

all

gro

ups

Eis

endra

th

(1988)

pp

Rel

atio

nsh

ipbet

wee

nin

tern

alan

d

exte

rnal

adm

inis

trat

ive

fact

ors

and

dir

ect

imple

men

tati

on

and

per

cepti

ons

of

use

fuln

ess

Gover

nm

enta

lag

enci

esin

Raj

asth

anS

tate

,In

dia

(n¼

16)

Eval

uat

ion

imple

men

tati

on;

dec

isio

nor

poli

cyse

ttin

g;

stak

ehold

erin

volv

emen

t

Poli

cym

aker

soft

enre

ject

edre

c-

om

men

dat

ions

of

eval

uat

ions

bec

ause

they

wer

enot

poli

tica

lly,

tech

nic

ally

,or

finan

cial

lyvia

ble

.

Hig

hle

vel

sof

use

wer

ere

late

dto

the

involv

emen

tof

hig

h-l

evel

exec

uti

ves

inth

ere

vie

wof

find-

ings,

form

ula

tion,

and

foll

ow

-up

of

reco

mm

endat

ions

for

acti

on.

Both

form

alan

din

form

al

adm

inis

trat

ive

arra

ngem

ents

wer

eim

port

ant

for

eval

uat

ion

use

.T

he

level

of

use

was

posi

-

tivel

yas

soci

ated

wit

hth

esa

l-

ience

of

apro

gra

mfo

rto

ple

vel

poli

cym

aker

s.B

yan

dla

rge,

hig

h-l

evel

poli

cym

aker

sco

nsi

d-

ered

the

eval

uat

ion

findin

gs

cred

ible

only

ifth

eyar

esu

p-

port

edby

oth

erso

urc

esof

info

rmat

ion

(conti

nued

)


396



Fors

set

al.

(1994)

pp

Explo

rero

leof

eval

uat

or

inorg

a-

niz

atio

nal

lear

nin

g;

rela

tionsh

ip

bet

wee

nqual

ity

of

report

and

eval

uat

ion

atti

tudes

wit

hco

gni-

tive

and

inst

rum

enta

luti

liza

tion

Norw

egia

nA

idA

dm

inis

trat

ion

Agen

cy(n¼

1)

Eval

uat

ion

imple

men

tati

on;

stak

ehold

erin

volv

emen

t

Alt

hough

aid

adm

inis

trat

ors

read

the

vas

tm

ajori

tyof

the

eval

ua-

tions

that

are

rele

van

tto

thei

r

posi

tions,

the

maj

ori

tyonly

lear

n

ali

ttle

from

the

report

s.S

uc-

cess

ful

lear

nin

gocc

urs

thro

ugh

two

pro

cess

es:

lear

nin

gth

rough

involv

emen

tan

dle

arnin

g

thro

ugh

com

munic

atio

n.

Involv

-

ing

adm

inis

trat

ors

inth

eco

nduct

of

eval

uat

ions

and

impro

vin

g

com

munic

atio

nof

eval

uat

ion

info

rmat

ion

wil

lm

axim

ize

org

a-

niz

atio

nal

lear

nin

gth

rough

eval

uat

ions

Gre

ene

(1987)

pp

Exam

ines

rela

tionsh

ipbet

wee

nty

pe

and

mea

nin

gfu

lnes

sof

stak

e-

hold

erpar

tici

pat

ion

and

use

Hum

anse

rvic

eag

enci

es(n¼

2)

Eval

uat

ion

imple

men

tati

on;

eval

uat

or

com

pet

ence

;st

ake-

hold

erin

volv

emen

t

Tw

opar

tici

pat

ory

eval

uat

ions

in

loca

lhum

anse

rvic

eag

enci

es

enco

ura

ged

stak

ehold

erpar

tici

-

pat

ion

inpla

nnin

gth

eev

aluat

ion.

Sta

keh

old

ers

report

edboth

con-

ceptu

alan

din

stru

men

tal

use

of

the

eval

uat

ion.

Inad

dit

ion,

sta-

keh

old

ers

also

found

sym

boli

c

way

sin

whic

hth

epro

cess

was

use

ful

(conti

nued

)


397



Gre

ene

(1988)

ppp

Inves

tigat

eth

ere

lati

onsh

ipbet

wee

n

com

munic

atio

nof

resu

lts

(pro

-

cess

,co

nte

nt,

and

par

tici

pat

ion

as

shar

eddec

isio

nm

akin

g)

and

uti

-

liza

tion

(conce

ptu

al,

inst

rum

en-

tal,

and

sym

boli

c)

Hum

anse

rvic

eag

enci

es(n¼

2)

Eval

uat

ion

imple

men

tati

on;

dec

isio

nor

poli

cyse

ttin

g;

stak

ehold

erin

volv

emen

t

Sta

keh

old

erte

amm

ember

sre

port

ed

occ

urr

ence

of

inst

rum

enta

l,co

n-

ceptu

al,

and

sym

boli

cuse

sar

is-

ing

from

both

the

eval

uat

ion

pro

cess

and

resu

lts.

The

foll

ow

-

ing

char

acte

rist

ics

of

the

eval

ua-

tion

report

ing

pro

cess

wer

e

bel

ieved

tohav

efa

cili

tate

duse

,

the

pro

cess

was

:ongoin

gan

d

iter

ativ

e;in

cluded

both

wri

tten

report

san

dst

akeh

old

ergro

up

dis

cuss

ions;

pre

sente

dth

ere

sult

s

com

pre

hen

sivel

yan

din

avar

iety

of

form

ats;

was

open

and

plu

ra-

list

ic;

and

was

tail

ore

dto

the

audie

nce

s.A

ddit

ional

ly,

stak

e-

hold

ers

wer

eac

tivel

yen

gag

edin

the

eval

uat

ion

and

com

munic

a-

tion

of

resu

lts,

and

the

eval

uat

or

funct

ioned

asan

advoca

tefo

ruse

duri

ng

and

afte

rth

eev

aluat

ion

was

conduct

ed

Had

dock

(1998)

pR

elat

ionsh

ipbet

wee

nle

gis

lati

ve

eval

uat

ion

char

acte

rist

ics

(com

-

mit

tee

type,

type

of

foll

ow

-up,

man

dat

eduse

,re

lati

onsh

ipw

ith

budget

ary

com

mit

tees

,an

d‘‘

fire

-

alar

mvs.

poli

ce-p

atro

l’’–

type

eval

uat

ions)

and

inst

rum

enta

luse

Sta

tele

gis

lati

ve

pro

gra

mev

aluat

ion

off

ices

(n¼

28)

Dec

isio

nor

poli

cyse

ttin

g;

stak

ehold

erin

volv

emen

t

Uti

liza

tion

dif

fere

nce

sap

par

entl

y

exis

tbet

wee

nth

efe

der

alan

d

stat

ele

vel

s.E

val

uat

ion

off

ices

in

stat

esw

ith

poli

cies

and

pro

ce-

dure

sm

andat

ing

reco

mm

enda-

tio

nar

esl

igh

tly

mo

reli

kel

yto

hav

ehig

her

imple

men

tati

on

rate

s

than

are

off

ices

inst

ates

wit

hno

such

poli

cies

.Par

tici

pat

ion

of

the

budget

com

mit

tee

inth

ese

lect

ion

of

topic

sfo

rpro

gra

mev

aluat

ions

does

not

nec

essa

rily

incr

ease

the

pro

bab

ilit

yof

eval

uat

ion

use

in

the

budget

dec

isio

n-m

akin

g

pro

cess

(conti

nued

)


398



Hopst

ock

etal

.

(1993)

pP

erce

ived

use

fuln

ess

of

eval

uat

ion

findin

gs

of

Tit

leV

IIbil

ingual

educa

tion

pro

gra

ms

Tit

leV

II-f

unded

educa

tion

pro

-

gra

ms

(n¼

18)

Dec

isio

nor

poli

cyse

ttin

gF

ewT

itle

VII

eval

uat

ion

report

s

wer

efe

der

ally

use

dfo

rth

efo

l-

low

ing

reas

ons.

The

purp

ose

san

d

audie

nce

sfo

rT

itle

VII

eval

ua-

tions,

asw

ell

asth

eev

aluat

ion

nee

ds

of

the

U.S

.D

epar

tmen

tof

Educa

tion

and

of

loca

lT

itle

VII

wer

enot

clea

rly

des

crib

edby

the

U.S

.D

epar

tmen

tof

Educa

tion.

Bec

ause

of

thei

rla

ckof

form

al

trai

nin

gin

eval

uat

ion

and

stat

is-

tics

and

bec

ause

of

the

larg

e

num

ber

of

pro

ject

sfo

rw

hic

hth

ey

are

resp

onsi

ble

,th

eO

ffic

eof

Bil

ingual

Educa

tion

and

Min

ori

ty

Lan

guag

esA

ffai

rsP

roje

ctO

ffi-

cers

wer

enot

able

toper

form

over

all

pro

gra

man

alyse

sor

to

pro

vid

edet

aile

dfe

edbac

kto

pro

ject

sab

out

thei

rev

aluat

ions

Johnso

n

(1993)

pp

Cre

ate

and

test

ath

eore

tica

lpro

cess

model

rela

ted

touse

;re

lati

onsh

ip

bet

wee

nle

vel

sof

par

tici

pat

ion,

com

pet

ing

info

rmat

ion,

truth

and

uti

lity

test

s,an

din

tere

sts

and

ideo

logy

and

expec

ted

level

of

uti

liza

tio

n

Eval

uat

ion

use

rsan

dpro

duce

rs

affi

liat

edw

ith

the

Geo

rgia

Inno-

vat

ion

Pro

gra

m(n¼

75)

Eval

uat

ion

imple

men

tati

on;

dec

isio

nor

poli

cyse

ttin

g;

stak

ehold

erin

volv

emen

t

Par

tici

pat

ion

inev

aluat

ion

was

most

likel

yin

org

anic

org

aniz

atio

ns,

com

pose

dof

chan

ge-

ori

ente

d

indiv

idual

s,w

ith

aper

son-

focu

sed

eval

uat

or.

Inst

rum

enta

l

uti

liza

tion

was

consi

der

edm

ost

likel

yin

situ

atio

ns

char

acte

rize

d

by

hig

hpar

tici

pat

ion,

affi

rmat

ive

truth

,an

duti

lity

test

ing,

and

when

inte

rest

san

did

eolo

gy

wer

e

support

ed.

Com

pet

ing

info

rma-

tion

was

notfo

und

tobe

rela

ted

to

inst

rum

enta

luti

liza

tion

(conti

nued

)


399



Johnst

on

(1986)

pp

Exam

ined

rela

tionsh

ips

bet

wee

n

type

of

eval

uat

ion

reco

mm

enda-

tions

and

acce

pta

nce

/use

or

like-

lihood

of

imple

men

tati

on

of

reco

mm

endat

ions

GA

Ore

port

s(n¼

176)

Eval

uat

ion

imple

men

tati

on;

dec

isio

nor

poli

cyse

ttin

g

Acc

epta

nce

of

GA

Ore

com

men

da-

tion

ishig

h.

Fac

tors

asso

ciat

ed

wit

hth

ehig

hac

cepta

nce

rate

incl

udes

that

the

reco

mm

enda-

tions

are

gen

eral

lyof

the

low

-

level

beh

avio

ral

com

pli

ance

–

type

and

the

stat

us

of

the

GA

Oas

afo

rmal

,fe

der

ally

man

dat

ed

outs

ide

eval

uat

ion

org

aniz

atio

n.

Addit

ional

ly,

the

met

hodolo

gic

al

qual

ity

of

the

studie

sco

ntr

ibute

s

toth

eir

uti

liza

tio

n

Laf

leur

(1995)

pp

Ret

rosp

ecti

ve

exam

inat

ion

of

one

school

dis

tric

t’s

par

tici

pat

ory

pro

gra

mev

aluat

ion

appro

ach

and

the

uti

liza

tion

of

eval

uat

ion

resu

lts

Can

adia

nsc

hool

dis

tric

t(n¼

1)

Eval

uat

ion

imple

men

tati

on;

dec

isio

nor

poli

cyse

ttin

g;

stak

ehold

erin

volv

emen

t

Bei

ng

involv

edin

the

eval

uat

ion

resu

lted

inth

epri

mar

yuse

rs

feel

ing

more

empow

ered

and

hav

ing

impro

ved

eval

uat

ion

skil

ls.

Quic

ker

turn

around

tim

e

on

resu

lts

would

impro

ve

use

.

Als

oim

port

ant

isa

support

ive

org

aniz

atio

nal

cult

ure

and

ongoin

g,

hig

h-q

ual

ity

com

munic

atio

n

Lee

and

Cou-

sins

(1995)

pp

Exam

ined

the

effe

cts

of

involv

e-

men

tin

apar

tici

pat

ory

eval

uat

ion

on

imple

men

ting

exte

rnal

ly

funded

,sc

hool-

dir

ecte

dch

ange

incl

udin

gth

eim

pac

tof

the

eva-

luat

ion

on

the

eval

uat

ion

consu

ltan

t

Can

adia

nsc

hools

(n¼

4)

Dec

isio

nor

poli

cyse

ttin

g;

stak

ehold

erin

volv

emen

t

Afo

undat

ion

pro

vid

edac

cess

toan

eval

uat

ion

consu

ltan

tto

four

schools

who

had

rece

ived

a

pro

gra

m-d

evel

opm

ent

gra

nt.

Eac

h

school

was

ata

dif

fere

nt

stag

ein

the

eval

uat

ion

but

none

had

yet

pro

duce

dan

yre

port

s.S

takeh

old

er

par

tici

pat

ion

allo

wed

for

gre

ater

under

stan

din

gab

out

eval

uat

ion.

Eval

uat

ions

wer

est

ill

inth

eea

rly

stag

es,

sono

report

sof

use

,but

eager

nes

san

den

thusi

asm

about

use

was

note

d

(conti

nued

)


400



Mal

en,

Mur-

phy,

and

Gea

ry

(1988)

pp

Anal

ysi

sof

the

effe

ctof

asp

ecif

ic

pro

gra

mev

aluat

ion

wit

ha

‘‘poli

tica

l’’

eval

uat

ion

report

and

uniq

ue

dec

isio

nco

nte

xt

and

the

exte

nt

of

acce

pta

nce

of

dat

a/re

c-

om

men

dat

ions

and

impac

tof

the

report

Uta

hst

ate

legis

latu

re,

inte

rvie

ws

wit

h21

indiv

idual

s(n¼

1/2

1)

Eval

uat

ion

imple

men

tati

on;

dec

isio

nor

poli

cyse

ttin

g

Char

acte

rist

ics

of

the

eval

uat

ion

and

conte

xt

inte

ract

edto

mak

e

the

eval

uat

ion

info

rmat

ion

a

‘‘si

gnif

ican

tth

reat

,’’

ath

reat

to

per

vas

ive

ideo

logie

s,poli

tica

l

alig

nm

ents

,re

form

com

mit

-

men

ts,

and

educa

tion

appro

pri

a-

tions.

The

eval

uat

ion

expose

d

div

ides

ina

frag

ile

coal

itio

nan

d

thre

aten

edco

nnec

tions

inth

e

legis

latu

re

Mar

ra(2

003)pppp

Use

of

eval

uat

ion

for

impro

vin

g

publi

corg

aniz

atio

ns’

per

for-

man

ceth

rough

bet

ter

des

ign

of

gover

nan

cest

ruct

ure

san

dm

ore

entr

epre

neu

rial

man

ager

ial

effo

rts

Worl

dB

ank

eval

uat

ion

studie

s

(n¼

4)

Eval

uat

ion

imple

men

tati

on;

dec

isio

nor

poli

cyse

ttin

g

Fiv

ekey

issu

esw

ere

iden

tifi

edas

affe

ctin

guse

:(a

)G

over

nan

ce

stru

cture

saf

fect

the

pote

nti

alfo

r

eval

uat

ion

topla

yas

ach

eck

and

bal

ance

wit

hin

the

org

aniz

atio

n

and

enfo

rce

resu

lts

acco

unta

bil

-

ity.

(b)

The

hig

h-p

rofi

lepoli

tica

l

role

of

the

eval

uat

ion

dep

artm

ent

hel

ps

eval

uat

ion

tobe

acce

pte

d

and

val

ued

for

stra

tegic

pla

nnin

g

atth

eap

exof

the

org

aniz

atio

n.

(c)

Man

ager

sdis

count

eval

uat

ion

for

thei

row

nw

ork

and

ascr

ibe

hig

her

sali

ence

for

thei

rsu

bord

i-

nat

es.

(d)

Most

inte

rvie

wee

s

endors

eth

esy

mboli

cro

leof

eval

uat

ion

tole

git

imiz

ea

posi

-

tion

or

dec

isio

n.

(e)

Act

ionab

le

and

evid

ence

-bas

edre

com

men

-

dat

ions

wer

eli

kel

yto

be

taken

into

acco

unt

(conti

nued

)


401



Mar

shan

d

Gla

ssic

k

(1988)

pE

ffec

tof

types

of

eval

uat

ion

rec-

om

men

dat

ions

(subje

ct,

audi-

ence

,sp

ecif

icit

y,

and

dep

th)

and

imple

men

tati

on

of

reco

mm

enda-

tions

by

schools

Eval

uat

ions

conduct

edby

eval

uat

ion

bra

nch

of

larg

em

etro

school

dis

tric

t(n¼

4)

Eval

uat

ion

imple

men

tati

on;

stak

ehold

erin

volv

emen

t

Sch

ool

adm

inis

trat

ors

use

dre

com

-

men

dat

ion

more

when

they

wer

e

det

aile

dan

dar

ose

from

ver

bal

dis

cuss

ions

bet

wee

nth

est

ake-

hold

ers

and

eval

uat

ors

.R

ecom

-

men

dat

ions

about

earl

yphas

esof

apro

ject

wer

em

ore

likel

yto

be

use

dif

they

focu

sed

on

inst

ruc-

tional

chan

ge;

reco

mm

endat

ions

from

late

rphas

esw

ere

more

likel

yto

be

use

dif

they

focu

sed

on

adm

inis

trat

ive

pro

ble

ms.

Ver

-

bal

inte

ract

ion

bet

wee

nth

eev

a-

luat

or

and

pro

gra

mst

aff

enhan

ced

the

under

stan

din

g,

acce

pta

nce

,an

duti

liza

tion

of

the

reco

mm

endat

ions

McC

orm

ick

(1997)

ppp

Rel

atio

nsh

ipbet

wee

nuse

rs’

com

-

mit

men

tto

the

pro

gra

m,

invol-

vem

ent

wit

hth

epro

gra

m,

atti

tude

tow

ard

eval

uat

ion,

org

aniz

atio

nal

posi

tion,

and

type

of

org

aniz

atio

nan

dre

port

ed

conce

ptu

al,

pro

cess

ing,

per

sua-

sive,

and

inst

rum

enta

luse

s

Pote

nti

alev

aluat

ion

use

rsof

the

pro

gra

mev

aluat

ion

div

isio

nof

a

stat

ele

gis

lati

ve

audit

or

and

a

soci

alse

rvic

ere

sear

chorg

aniz

a-

tio

n(n¼

89)

Dec

isio

nor

poli

cyse

ttin

g;

stak

ehold

ersi

nvolv

emen

t

Conce

ptu

aluse

exce

eded

all

oth

er

types

of

use

,an

dpro

cess

ing

use

exce

eded

per

suas

ive

or

inst

ru-

men

tal

use

.In

volv

emen

tin

eva-

luat

ion

was

hig

hly

rela

ted

toal

l

types

of

use

,es

pec

iall

y‘‘

pro

-

cess

ing

use

.’’

Publi

c/gover

nm

ent

and

pri

vat

enonpro

fit

org

aniz

a-

tions

uti

lize

dev

aluat

ion

info

r-

mat

ion

equal

ly.

Man

ager

sw

ere

more

acti

ve

than

legis

lato

rsin

term

sof

pro

cess

ing

use

.A

ttit

ude

tow

ard

eval

uat

ion

had

litt

le

infl

uen

ceon

eval

uat

ion

use (conti

nued

)


402



New

man

etal

.

(1987)

pp

Eff

ect

of

confl

ict,

import

ance

,se

t-

ting,

and

super

inte

nden

tsu

pport

on

dec

isio

nm

akin

gas

mea

sure

d

by

Dec

isio

n-M

akin

gIn

form

atio

n

Nee

ds

Sca

le(N

ewm

an,

Bro

wn,

Riv

ers,

&G

lock

,1983)

Sch

ool

boar

dm

ember

s(n¼

361)

Dec

isio

nor

poli

cyse

ttin

gE

val

uat

ion

use

isin

fluen

ced

by

the

per

ceiv

edim

port

ance

and

sett

ing

of

the

pro

gra

m.

When

mak

ing

a

dec

isio

nab

out

apro

gra

mof

hig

h

import

ance

,boar

dm

ember

s

requir

edm

ore

tim

e,m

ore

info

r-

mat

ion,

and

more

conta

cts

wit

ha

consu

ltan

t.P

rogra

mse

ttin

ghad

the

gre

ates

tst

rength

of

asso

cia-

tion.

Pro

gra

mco

nfl

ict

infl

uen

ced

info

rmat

ion

nee

ds.

When

the

pro

gra

mw

asof

hig

hco

nfl

ict(a

nd

no

know

ledge

or

super

inte

nden

t

atti

tude

was

giv

en),

boar

dm

em-

ber

sw

ante

dm

ore

tim

e,m

ore

info

rmat

ion,

more

per

sonal

con-

tact

s,an

dco

nta

cts

wit

hco

nsu

l-

tants

com

par

edto

low

confl

ict

sett

ings

Pott

s(1

998)

pp

Rel

atio

nsh

ipbet

wee

nev

aluat

ion

met

hod

(quan

tita

tive,

qual

itat

ive,

or

mix

ed)

and

conce

ptu

alan

d

inst

rum

enta

luse

Ten

adm

inis

trat

ors

from

studen

t

serv

ice

pro

gra

ms

ata

larg

est

ate

univ

ersi

ty(n¼

10)

Eval

uat

ion

imple

men

tati

on

Univ

ersi

tyad

min

istr

ators

felt

that

the

findin

gs

from

mix

ed-m

ethod

report

spro

duce

dgre

ater

know

l-

edge

gai

n,

wer

em

ore

cred

ible

,

and

wer

em

ore

use

fulth

ansi

ngle

-

met

hod

quan

tita

tive

or

qual

ita-

tive

studie

s

Pre

skil

lan

d

Car

acel

li

(1997)

ppp

Eval

uat

ors

’bel

iefs

on

eval

uat

ion

use

,in

cludin

gth

eim

pli

cati

ons

of

stak

ehold

erin

volv

emen

ton

use

Am

eric

anE

val

uat

ion

Ass

oci

atio

n

(AE

A)

eval

uat

ion

use

dby

Topi-

cal

Inte

rest

Gro

up

(TIG

)m

em-

ber

s(n¼

282)

Sta

keh

old

erin

volv

emen

tS

urv

eyof

eval

uat

ors

’per

cepti

ons

of

eval

uat

ion

use

.Iden

tifi

edse

ven

most

import

ant

stra

tegie

sto

infl

uen

ceuse

:pla

nnin

gfo

ruse

at

beg

innin

gof

eval

uat

ion,

iden

ti-

fyin

gan

dpri

ori

tizi

ng

inte

nded

use

rsan

duse

s,des

ignin

g

eval

uat

ion

wit

hli

mit

ed

reso

urc

es,

pla

nnin

gfo

rco

mm

u-

nic

atin

gw

ith

stak

ehold

ers

thro

ughout.

Found

that

def

init

ion

of

use

has

expan

ded

from

trad

i-

tional

toin

clude

pro

cess

use

and

org

aniz

atio

nal

lear

nin

gco

nce

pts

(conti

nued

)


403



Rin

ne

(1994)

ppp

Rel

atio

nsh

ipam

ong

the

per

ceiv

ed

purp

ose

of

eval

uat

ion

(pro

gra

m

impro

vem

ent,

judge

mer

it/w

ort

h,

know

ledge

gen

erat

ion)

and

the

uti

liza

tion

of

eval

uat

ion

resu

lts

takin

gin

toac

count

anxie

tyle

vel

of

pote

nti

alen

duse

rsof

eval

uat

ion

Hea

lth

care

educa

tors

who

teac

h

hea

lth

pro

moti

on

and

pre

ven

tion

pro

gra

ms

(n¼

540)

Dec

isio

nor

poli

cyse

ttin

gT

he

study

found

the

import

ance

of

exte

rnal

purp

ose

sfo

rco

nduct

ing

eval

uat

ion

asco

mpar

edto

inte

r-

nal

purp

ose

s.E

xte

rnal

purp

ose

s

pre

dic

ted

posi

tive

mai

nte

nan

ce

and

neg

ativ

ech

ange.

Both

inte

r-

nal

and

exte

rnal

purp

ose

spre

-

dic

ted

conce

ptu

aluse

.W

ith

the

exce

pti

on

of

no

use

,all

const

ruct

s

of

use

wer

epre

dic

ted

by

one

or

more

of

the

anxie

tyco

nst

ruct

s.

When

contr

oll

ing

for

anxie

ty,

no

consi

der

able

incr

ease

inpre

dic

t-

abil

ity

was

found

for

the

asso

ci-

atio

nbet

wee

npurp

ose

and

use

Rock

wel

l

etal

.

(1990)

pp

Exam

ined

the

impac

tof

atte

ndin

gto

Pat

ton’s

(1997)

uti

liza

tion-

focu

sed

eval

uat

ion

‘‘per

sonal

fact

ors

’’on

eval

uat

ion

use

s

One

team

of

four

exte

nsi

on

staf

f

who

wer

ein

tended

eval

uat

ion

use

rs(n¼

1/4

)

Eval

uat

ion

imple

men

tati

on;

dec

isio

nor

poli

cyse

ttin

g;

stak

ehold

erin

volv

emen

t

Six

fact

ors

wer

eid

enti

fied

as

enco

ura

gin

gev

aluat

ion

use

afte

r

atte

ndin

gto

the

per

sonal

fact

or

in

the

pla

nnin

gof

the

eval

uat

ion:(a

)

the

inte

nded

use

r’s

info

rmat

ion

nee

ds;

(b)

the

tim

elin

ess

of

the

study;

(c)

the

inte

nded

use

r’s

ow

ner

ship

of

the

info

rmat

ion

that

was

fost

ered

by

thei

rin

volv

e-

men

t;(d

)in

tera

ctio

nam

ong

inte

nded

use

rsan

dth

eev

aluat

or;

(e)

the

eval

uat

ion’s

met

hodolo

-

gic

alap

pro

pri

aten

ess

and

qual

ity;

and

(e)

dis

cuss

ion

of

the

resu

lts

in

stee

ring

com

mit

tee

mee

tings

(conti

nued

)


404



San

thiv

eera

n

(1995)

pIm

pac

tof

eval

uat

ion

type,

inte

rnal

fact

ors

,an

dex

tern

alfa

ctors

on

five

dom

ains

of

use

mea

sure

dby

Kir

khar

tan

dG

lass

er(1

991)

Use

Sca

le

Men

tal

hea

lth

exec

uti

ve

dir

ecto

rs

and

pro

gra

mad

min

istr

ators

(n¼

180)

Dec

isio

nor

poli

cyse

ttin

gT

he

key

fact

ors

found

affe

ctin

gth

e

use

of

eval

uat

ion

dat

aw

ere

the

pro

port

ion

of

the

budget

allo

-

cate

dfo

rev

aluat

ions,

the

avai

l-

abil

ity

of

anev

aluat

ion

dir

ecto

r,

and

the

pro

port

ion

of

fundin

g

from

stat

ean

dlo

cal

sourc

es.

Per

sonal

char

acte

rist

ics

(gen

der

,

age,

and

ethnic

ity)

and

job-

rela

ted

char

acte

rist

ics

(tim

esp

ent

inper

sonnel

man

agem

ent,

super

-

vis

ion,

and

pro

gra

mdev

elop-

men

t)w

ere

found

tobe

pote

nti

al

pre

dic

tors

of

eval

uat

ion

uti

liza

-

tion.

The

atti

tudes

of

the

indi-

vid

ual

resp

onden

tsto

war

d

eval

uat

ion

wer

enot

rela

ted

to

eval

uat

ion

uti

liza

tion

Shea

(1991)

ppp

Rel

atio

nsh

ipbet

wee

nev

aluat

ion

pro

cess

,ev

aluat

or

char

acte

rist

ics,

and

the

dec

isio

nco

nte

xt

and

conce

ptu

al,

inst

rum

enta

l,an

d

sym

boli

cuse

mea

sure

dby

item

s

from

Johnso

n(1

980)

and

Wee

k

(1979)

Can

adia

nE

val

uat

ion

Soci

ety

mem

ber

s(n¼

332)

Eval

uat

ion

imple

men

tati

on;

dec

isio

nor

poli

cyse

ttin

g;

eval

uat

or

com

pet

ence

;

stak

ehold

erin

volv

emen

t

Can

adia

nev

aluat

ors

report

edhig

h

level

sof

use

(91–99%

)of

the

last

eval

uat

ion.M

ost

of

the

use

sw

ere

conce

ptu

al,

foll

ow

edby

inst

ru-

men

tal

and

per

suas

ive

use

s.

Com

ple

xre

lati

onsh

ips

exis

ted

bet

wee

nth

ree

cate

gori

esof

indep

enden

tvar

iable

s(p

roce

ss,

eval

uat

or,

and

conte

xt)

and

use

.

All

thre

eca

tegori

esof

fact

ors

had

som

ere

lati

onsh

ipw

ith

inst

ru-

men

tal

and

conce

ptu

aluse

.P

er-

suas

ive

use

was

only

asso

ciat

ed

wit

hone

pro

cess

and

two

eva-

luat

or

var

iable

s.T

he

num

ber

of

conta

cthours

spen

tin

any

of

the

foll

ow

ing

acti

vit

ies

was

signif

i-

cantl

yas

soci

ated

wit

hin

stru

-

men

tal

use

:pla

nnin

g,

imple

men

tati

on,

and

dis

sem

inat

ion

(conti

nued

)


405



Sle

ezer

(1987)

pE

ffec

tof

types

of

eval

uat

ion

report

s

(info

rmat

ional

,ex

amin

atio

nal

,or

anal

yti

cal)

on

level

of

finan

cial

support

and

logic

of

budget

dec

isio

nm

akin

g

Dec

isio

nm

aker

sin

man

ufa

cturi

ng

org

aniz

atio

ns

resp

onsi

ble

for

finan

cial

reso

urc

eal

loca

tion

for

trai

nin

g(n¼

40)

Eval

uat

ion

imple

men

tati

on

Eval

uat

ion

report

shad

aver

ylo

w

level

of

infl

uen

ceover

dec

isio

ns

about

fundin

gtr

ainin

gpro

gra

ms.

Only

50%

of

the

resp

onden

ts

even

looked

atth

ere

port

pri

or

to

mak

ing

adec

isio

nan

dth

ose

who

read

the

report

did

not

use

it,

did

not

bel

ieve

it,

or

rela

ted

itto

a

pre

vio

us

pro

gra

monly

.N

ore

la-

tionsh

ipw

asfo

und

bet

wee

nty

pe

of

report

and

its

use

Sper

lazz

a

(1995)

pp

Des

crib

eth

eim

pac

tof

par

tici

pat

ion

of

ate

amof

eval

uat

ion

pro

fes-

sional

son

thei

rpro

fess

ional

dev

elopm

ent

and

use

of

resu

lts

An

eval

uat

ion

team

,fo

ur

mem

ber

s

(n¼

1/4

)

Sta

keh

old

erin

volv

emen

tP

arti

cipat

ory

eval

uat

ion

was

seen

as

advan

tageo

us

and

par

tici

pat

ion

was

rela

ted

toin

crea

sed

use

.

Advan

tages

of

the

appro

ach

incl

uded

the

team

get

ting

to

know

thei

rco

llea

gues

and

gai

n

under

stan

din

gof

thei

rpro

gra

m,

val

uin

gst

aff

involv

emen

tin

dec

isio

nm

akin

g,

and

buil

din

ga

sense

of

ow

ner

ship

of

thei

r

pro

gra

m

Turn

bull

(1999)

pp

Tes

tca

usa

lre

lati

onsh

ips

inpro

-

pose

dm

odel

bet

wee

npar

tici

pa-

tory

eval

uat

ion

char

acte

rist

ics

and

use

of

eval

uat

ion

info

rmat

ion

Tea

cher

sfr

om

Bri

tish

Colu

mbia

school

accr

edit

atio

npro

gra

m

(n¼

315)

Sta

keh

old

erin

volv

emen

tH

igh

level

sof

infl

uen

cew

ere

rela

ted

tohig

hle

vel

sof

par

tici

-

pat

ion

effi

cacy

.T

her

ew

asa

posi

tive

rela

tionsh

ipbet

wee

n

par

tici

pat

ion

effi

cacy

and

inst

ru-

men

tal

and

sym

boli

cuse

,su

g-

ges

ting

that

par

tici

pat

ion

effi

cacy

isa

med

iati

ng

fact

or

linkin

g

acti

on

theo

ry(p

arti

cipat

ion)

and

conce

ptu

alth

eory

(use

)

(conti

nued

)


406



Wei

sset

al.

(2005)

pp

Exam

inat

ion

of

the

use

and

infl

u-

ence

of

DA

RE

pro

gra

mev

alua-

tions;

appli

cati

on

of

Mar

kan

d

Hen

ry’s

(2004)

mec

han

ism

sof

eval

uat

ion

infl

uen

ce

Law

enfo

rcem

ent

off

icia

lsan

d

school

dis

tric

tad

min

istr

ators

from

16

com

munit

ies

wit

han

d

wit

hout

DA

RE

eval

uat

ions

(n¼

128)

Eval

uat

ion

imple

men

tati

on;

dec

isio

nor

poli

cyse

ttin

g

DA

RE

eval

uat

ions

wer

euse

din

a

var

iety

of

way

s:poli

tica

lly

to

per

suad

eoth

ers,

inst

rum

enta

lly

tom

ake

dec

isio

ns

about

futu

re

pro

gra

mm

ing,

and

conce

ptu

ally

inte

rms

of

ara

isin

gth

eco

n-

scio

usn

ess

of

stak

ehold

ers.

A

new

type

of

use

was

iden

tifi

ed,

‘‘im

pose

duse

’’in

whic

hdis

tric

ts

wer

efo

rced

tore

pla

ceth

epro

-

gra

mw

ith

one

on

agover

nm

ent

appro

ved

list

.T

he

pat

hw

ays

to

whic

hin

fluen

cew

asac

hie

ved

wer

eta

ngle

d,

com

ple

x,

and

dif

-

ficu

ltto

dis

cern

retr

osp

ecti

vel

y.

Moti

vat

ional

fact

ors

pla

yed

a

par

t;in

centi

ves

push

eddis

tric

ts

toap

ply

eval

uat

ion

resu

lts.

Addit

ional

ly,

the

urg

eto

act

rati

onal

lyin

fluen

ced

beh

avio

ral

use

of

the

resu

lts


407



References

Alkin, M.C., & Taut, S.M. (2003). Unbundling evaluation use. Studies in Educational Evaluation, 29, 1-12.

Altschuld, J. W., Yoon, J. S., & Cullen, C. (1993). The utilization of needs assessment results. Evaluation and

Program Planning, 16, 279-285.

Ayers, T. D. (1987). Stakeholders as partners in evaluation: A stakeholder-collaborative approach. Evaluation and


Bamberger, M. (2004). Influential evaluations: Evaluations that improved performance and impacts of development

programs. Washington, DC: Operations Evaluation Department, The World Bank.

Barrios, N. B. (1986). Utilization of evaluation information: A case study approach investigating factors related to

evaluation utilization in a large state agency. Dissertation Abstracts International: Section A: The Humanities and

Social Sciences, 47, 1704 (UMI 8616880).

Bober, C. E., & Bartlett, K. R. (2004). The utilization of training program evaluation in corporate universities. Human

Resource Development Quarterly, 15, 363-383.

Boyer, J. F., & Langbein, L. I. (1991). Factors influencing the use of health evaluation research in Congress.

Evaluation Review, 18, 507-532.

Brown-McGowan, S. (1992). Effects of decision maker and context variables on evaluation utilization. Dissertation

Abstracts International: Section A: The Humanities and Social Sciences, 53, 2261 (UMI 9226505).

Cai, M. (1996). An empirical examination of participatory evaluation: Teachers’ perceptions of their involvement and

evaluation use. Dissertation Abstracts International: Section A: The Humanities and Social Sciences, 57, 1921

(UMI 9629749).

Callahan, C. M., Tomlinson, C. A., Hunsaker, S. L., Bland, L. C., & Moon, T. (1995). Instruments and evaluation

designs used in gifted programs. The National Research Center on the Gifted and Talented, The University of

Virginia. Research Report: RM-95132.

Carpinello, S. E. (1989). The effect of power, consequence, and experience on nurse decision-makers’ utilization of

evaluation information. State University of New York at Albany. Dissertation Abstracts International: Section B:

The Sciences & Engineering, 50, 3395.

Chin, M. C. (2003). An investigation into the impact of using poetry and cartoons as alternative representational forms

in evaluation reporting. Dissertation Abstracts International: Section A: The Humanities and Social Sciences, 64,

394 (UMI 3081434).

Christie, C. A. (2007). Reported influence of evaluation data on decision makers’ actions: An empirical examination.

American Journal of Evaluation, 28, 8-25.

Combs, W. L. A. (1999). The predictive validity of teachers’ attitudes towards inclusive education on the conceptual

use of program evaluation information. Dissertation Abstracts International: Section A: The Humanities and

Social Sciences, 60, 3208 (UMI 9946401).

Cousins, J. B. (1995). Assessing program needs using participatory evaluation: A comparison of high and marginal

success cases. In J. B. Cousins & L. M. Earl (Eds.), Participatory evaluation in education: Studies in evaluation

use and organizational learning (pp. 55-71). London: Routledge.

Cousins, J. B. (1996). Consequences of researcher involvement in participatory evaluation. Studies in Educational

Evaluation, 22, 3-27.

Cousins, J.B. (2003). Utilization effects of participatory evaluation. In T. Kelleghan, & D. L. Stufflebeam (Eds.),

International handbook of educational evaluation (pp. 245-265). Great Britain: Kluwer Academic Publishers.

Cousins, J. B., Goh, S. C., Clark, S., & Lee, L. E. (2004). Integrating evaluative inquiry into the organizational culture:

A review and synthesis of the knowledge base. Canadian Journal of Program Evaluation, 19, 99-141.

Cousins, J. B., & Leithwood, K. A. (1986). Current empirical research on evaluation utilization. Review of

Educational Research, 56, 331-364.

Crotti, J. G. (1993). Evaluation utilization: A study of administrators’ perceptions of the uses of the long-range plan

evaluation process in Pennsylvania. Dissertation Abstracts International: Section A: The Humanities and Social

Sciences, 54, 4315 (UMI 9414267).

Earl, L. M. (1995). District-wide evaluation of school improvement: A system partners approach. In J. B. Cousins & L.

M. Earl (Eds.), Participatory evaluation in education: Studies in evaluation use and organizational learning

(pp. 21-32), London: Routledge.

Eisendrath, A. (1988). The use of development project evaluation information: A study of state agencies in India.

Dissertation Abstracts International: Section A: The Humanities and Social Sciences, 49, 1572 (UMI 8810460).

Forss, K., Cracknell, B., & Samset, K. (1994). Can evaluation help an organization to learn? Evaluation Review, 18,

574-591.

Greene, J. C. (1987). Stakeholder participation in evaluation design: Is it worth the effort? Evaluation and Program

Planning, 10, 379-394.




Greene, J. C. (1988). Communication of results and utilization in participatory program evaluation. Evaluation and


Guarino, C., Santibanez, L., Daley, G., & Brewer, D. (2004, May). A review of the research literature on teacher

recruitment and retention. Technical report TR-164-EDU. Santa Monica, CA: RAND.

Guba, E. G., & Lincoln, Y. S. (1989). Fourth generation evaluation. Newbury Park, CA: Sage.

Haddock, R. E. (1998). State legislative program evaluation: An assessment of recent claims of direct utilization in the

states. Dissertation Abstracts International: Section A: The Humanities and Social Sciences, 60, 881 (UMI

9921400).

Henry, G. T., & Mark, M. M. (2003a). Beyond use: Understanding evaluation’s influence on attitudes and actions.

American Journal of Evaluation, 24, 293-314.

Henry, G. T., & Mark, M. M. (2003b). Toward an agenda for research on evaluation. New Directions for Evaluation,

97, 69-80.

Hofstetter, C. H., & Alkin, M. C. (2003). Evaluation use revisited. In T. Kelleghan & D. L. Stufflebeam (Eds.),

International handbook of educational evaluation (pp. 197-222). Great Britain: Kluwer Academic Publishers.

Hopstock, P., Young, M., & Zehler, A. (1993). Serving different masters: Title VII evaluation practice and policy.

Vol. I – Final report. Arlington, VA: Development Associates IncReport: ED/OPP93-32.

Johnson, K.W. (1980). Academia and practice. Knowledge: Creation, Diffusion, Utilization, 2, 237-261.

Johnson, R. B. (1993). An exploratory conjoint measurement study of selected variables related to innovative

educational evaluation participation and instrumental utilization. Dissertation Abstracts International: Section A:

The Humanities and Social Sciences, 55, 64 (UMI 9416264).

Johnston, W. P., Jr. (1986). A study of the acceptance of management performance evaluation recommendations by

federal agencies: Lessons from GAO reports issued in FY 1983. Dissertation Abstracts International: Section A:

The Humanities and Social Sciences, 48, 2157 (UMI 8725055).

King, J.A. & Pechman, E.M. (1984). Pinning a wave to the shore: Conceptualizing evaluation use in school systems.

Educational Evaluation and Policy Analysis, 6, 241-451.

Kirkhart, K. E. (2000). Reconceptualizing evaluation use: An integrated theory of influence. New Directions for

Evaluation, 88, 5-23.

Kirkhart, K.E., Morgan, R.O., & Sincavage, J. (1991). Assessing evaluation performance and use: Test-Retest.

Evaluation Review, 15(4), 482-502.

Lafleur, C. (1995). A participatory approach to district-level program evaluation: The dynamics of internal evaluation.

In J. B. Cousins & L. M. Earl (Eds.), Participatory evaluation in education: Studies in evaluation use and

organizational learning (pp. 33-54). London: Falmer.

Lee, L. E., & Cousins, J. B. (1995). Participation in evaluation of funded school improvement: Effects and supporting

conditions. In J. B. Cousins & L. M. Earl (Eds.), Participatory evaluation in education: Studies in evaluation use

and organizational learning (pp. 72-85). London: Routledge.

Leviton, L.C., & Hughes, E.F.X. (1981). Research on the utilization of evaluations: A review and synthesis. Evalua-

tion Review, 5, 525-549.

Malen, B., Murphy, M. J., & Geary, S. (1988). The role of evaluation information in legislative decision making:

A case study of a loose cannon on deck. Theory into Practice, 27, 111-125.

Mark, M.M., Henry, G.T., & Julnes, G. (2000). Evaluation: An integrated framework for understanding, guiding, and

improving policies and programs. San Francisco: Jossey-Bass, Inc.

Mark, M. M., & Henry, G. T. (2004). The mechanisms and outcomes of evaluation influence. Evaluation, 10, 35-57.

Marra, M. (2003). Dynamics of evaluation use as organizational knowledge: The case of the World Bank. Dissertation


Marsh, D. D., & Glassick, J. M. (1988). Knowledge utilization in evaluation efforts: The role of recommendations.

Knowledge, 9, 323-341.

McCormick, E. R. (1997). Factors influencing the use of evaluation results. Dissertation Abstracts International:

Section A: The Humanities and Social Sciences, 58, 4187 (UMI 9815051).

Newman, D., Brown, R., Rivers, L., & Glock, R. (1983). School boards’ and administrators’ use of evaluation

information: Influencing factors. Evaluation Review, 7(1), 110-125.

Newman, D. L., Brown, R. D., & Rivers, L. (1987). Factors influencing the decision-making process: An examination

of the effect of contextual variables. Studies in Educational Evaluation, 13, 199-209.

Nunneley, R. D. (2008). The danger of theorizing under the influence: An analysis of the arguments in Henry and

Mark (2003) and Mark and Henry (2004). Unpublished manuscript. Minneapolis, MN: University of Minnesota.

Patton, M. Q. (1997). Utilization-focused evaluation. Thousand Oaks, CA: Sage.

Potts, S. A. K. (1998). Impact of mixed method designs on knowledge gain, credibility, and utility of program

evaluation findings. Dissertation Abstracts International: Section A: The Humanities and Social Sciences, 59,

1942 (UMI 9837695).




Preskill, H., & Caracelli, V. (1997). Current and developing conceptions of use: Evaluation use TIG survey results.

Evaluation Practice, 18, 209-225.

Rinne, C. (1994). The impact of anxiety as a mediating variable on health educators’ utilization of evaluation results.

Dissertation Abstracts International: Section B: The Sciences & Engineering, 54, 3554.

Rockwell, S. K., Dickey, E. C., & Jasa, P. J. (1990). The personal factor in evaluation use: A case study of a steering

committee’s use of a conservation tillage survey. Evaluation and Program Planning, 13, 389-394.

Russon, C., & Koehly, L. (1995). Construction of a scale to measure the persuasive impact of qualitative and quanti-

tative evaluation reports. Evaluation and Program Planning, 18(2), 165-177.

Santhiveeran, J. (1995). Factors influencing the utilization of evaluation findings in mental health centers: A national

survey. Dissertation Abstracts International: Section A: The Humanities and Social Sciences, 56, 3311

(UMI 9539597).

Scriven, M. (2007). Activist evaluation. Journal of MultiDisciplinary Evaluation, 4(7), i-ii.

Shea, M. P. (1991). Program evaluation utilization in Canada and its relationship to evaluation process, evaluator and

decision context variables. University of Windsor (Canada). Dissertation Abstracts International: Section B: The

Sciences & Engineering, 53, 597.

Shulha, L. M., & Cousins, J. B. (1997). Evaluation use: Theory, research, and practice since 1986. Evaluation

Practice, 18, 195-208.

Sleezer, C. M. (1987). The relationship between types of evaluation reports and support for the training function by

corporate managers. Project number twenty-one. Department of Vocational and Technical Education, University

of Minnesota.

Sperlazza, J. (1995). Involving school professionals in program evaluation in an urban school district. Dissertation


Turnbull, B. (1999). The mediating effect of participation efficacy on evaluation use. Evaluation and Program

Planning, 22, 131-140.

Weeks, E.C. (1979). The managerial use of evaluation findings. In H.C. Schulberg & J.M. Jerrell (Eds.), The evaluator

and management (pp. 137-156). Beverly Hills, CA: Sage.

Weiss, C. H., Murphy-Graham, E., & Birkeland, S. (2005). An alternate route to policy influence: How evaluations

affect DARE. American Journal of Evaluation, 26, 12-30.




Research on Evaluation Use - Western Michigan University

Documents

Transcript of Research on Evaluation Use - Western Michigan University