The Evaluation Checklist
23
The Evaluation Checklist Wes Martz, Ph.D.
description
Advantages and disadvantages of evaluation checklists and how to use them to improve evaluation practice. Presented at USF Center for Research, Evaluation, Assessment, and Measurement.
Transcript of The Evaluation Checklist
The Evaluation Checklist
Wes Martz, Ph.D.
2
A mnemonic device that consists of a list of activities, items, and
criteria used to perform a certain task.
3
Guidance for the collection of relevant evidence used to
determine the merit, worth, or significance of an evaluand.
4
Existing Evaluation Checklists
• Evaluation management • Evaluation models • Evaluation values and criteria • Metaevaluation • Evaluation capacity building • Checklist creation
http://www.wmich.edu/evalctr/checklists
5
Checklist Organizer
•Party planner lists
•Grocery lists
•Travel lists
Taxonomic
•Pre-flight checklist
•Product assembly instructions
•Medical procedures
Procedural
•Personnel selection qualities
•Rating sheets in gymnastics competition
•Sections of evaluation checklists
Evaluative
6
Good Checklists
Practical
Parsimonious
Precise
7
Criteria to Evaluate Checklists
• Applicability to full range of intended uses• Clarity• Comprehensiveness• Concreteness• Ease of use• Fairness• Parsimony• Pertinence to the content area
Stufflebeam, D. L. (2000) Guidelines for developing evaluation checklists: The checklist development checklist. Available at http://www.wmich.edu/evalctr/checklists/guidelines_cdc.pdf
8
Checklist Advantages
• Consolidate vast knowledge in a parsimonious manner• Improve task performance• Reduce influence of halo and Rorschach effects• Reduce resource use• Improve memory recall• Set out minimum necessary steps in a process
Scriven, M. (2005). Checklists. In S. Mathison (Ed.), Encyclopedia of evaluation (pp. 53-59). Thousand Oaks, CA: Sage.
9
Checklist Disadvantages
• Evaluation myopia• Inappropriate use• Fatigue resulting from overuse• Unnecessary complexity decreases reliability• Burdensome process delays completing evaluation
Martz, W. (in press). Validating an evaluation checklist using a mixed method design. Evaluation and Program Planning (2009). doi: 10.1016/j.evalprogplan.2009.10.005
10
Organizational Effectiveness Evaluation Checklist (OEC)
Developing and Validating an Evaluation Checklist: A Case Example
11
OEC Overview
• Organizational evaluation process framework• Iterative, explicit, weakly sequential• Six steps, 29 checkpoints• Criteria of merit checklist
– 12 universal criteria of merit– 84 suggested measures
12
OEC Overview
Step 1
Establish the boundaries of the evaluation
Step 2
Conduct a performance needs assessment
Step 3
Define the criteria to be used for the evaluation
Step 4
Plan and implement evaluation
Step 5
Synthesize performance data with values
Step 6
Communicate and report evaluation
activities
13
Criteria to Evaluate the OEC
• Applicability to full range of intended uses• Clarity• Comprehensiveness• Concreteness• Ease of use• Fairness• Parsimony• Pertinence to the content area
14
OEC Validation Process
• Phase 1: Expert panel review– Critical feedback survey– Written comments made on checklist
• Phase 2: Field test– Single-case study– Semi-structured interview
15
Expert Panel Overview
• Study participants– Subject matter experts (organizational and evaluation theorists)– Targeted users (professional evaluators, organizational
consultants, managers)
• Review OEC for providing critical feedback• Identify strengths and weaknesses• Complete the critical feedback survey• Write comments directly on the checklist
16
Expert Panel Data Analyses
• Critical feedback survey– Descriptive statistics– Parametric and nonparametric analysis of variance
• Written comments on checklist– Hermeneutic interpretation– Thematic analysis to cluster and impose meaning– Triangulation across participants to corroborate or falsify the
imposed categories
17
Key Findings
• Pertinent to content area• Clear• Fair• Sound theory
• Parsimony and ease of use were identified as areas to address
Content relevance Representativeness Substantive validity
18
Field Test Overview
• Conducted evaluation using revised OEC• Strictly followed the OEC to ensure fidelity• Post-eval semi-structured client interview• A formative metaevaluation to detect and
correct deficiencies in the process
19
Observations from Field Test
• Structured format minimized “scope-creep”• Identified several areas to clarify in OEC• Reinforced need for multiple measures,
transparency in standards• Minimal disruption to the organization
20
Validity Study Assessment
• Strengths– Relatively quick validation process– Based on relevant evaluative criteria– Features a real-world application
• Weaknesses – Single-case field study– Selection of the case study– Selection of the expert panel members
Martz, W. (in press). Validating an evaluation checklist using a mixed method design. Evaluation and Program Planning (2009). doi: 10.1016/j.evalprogplan.2009.10.005
21
Lessons Learned
• Checklist development should address unique attributes of the evaluand
• Sampling frame is critical• Checklist validation should be grounded in theory,
practice, and use• Mixed method approach provides increased confidence
in validation conclusions• All checklists are a “work-in-process”
22
Additional Checklist Resources
• Martz, W. (in press). Validating an evaluation checklist using a mixed method design. Evaluation and Program Planning (2009).
• Scriven, M. (2007). The logic and methodology of checklists. Available at http://www.wmich.edu/evalctr/checklists/papers/logic&methodology_dec07.pdf.
• Stufflebeam, D. L. (2000) Guidelines for developing evaluation checklists: The checklist development checklist. Available at http://www.wmich.edu/evalctr/checklists/guidelines_cdc.pdf.
• Stufflebeam, D. L. (2001). Evaluation checklists: Practical tools for guiding and judging evaluations. American Journal of Evaluation, 22, 71–79.
http://www.wmich.edu/evalctr/checklists
