Theory-Driven Evaluation for Assessing and Improving · PDF file ·...
Transcript of Theory-Driven Evaluation for Assessing and Improving · PDF file ·...
![Page 1: Theory-Driven Evaluation for Assessing and Improving · PDF file · 2011-02-20Theory-Driven Evaluation for Assessing and Improving ... and professors • Interventions ... Education/signage](https://reader031.fdocuments.net/reader031/viewer/2022022500/5aa4a7c67f8b9a7c1a8c6b3a/html5/thumbnails/1.jpg)
Theory-Driven Evaluation for Assessing andImproving Planning, Implementation, and
Effectiveness
Huey T. Chen, Ph.D.Center for Disease Control and Prevention
E-mail: [email protected]
DISCLAIMER: The views expressed in this presentatio n do not necessarily represent the views of CDC.
![Page 2: Theory-Driven Evaluation for Assessing and Improving · PDF file · 2011-02-20Theory-Driven Evaluation for Assessing and Improving ... and professors • Interventions ... Education/signage](https://reader031.fdocuments.net/reader031/viewer/2022022500/5aa4a7c67f8b9a7c1a8c6b3a/html5/thumbnails/2.jpg)
Ground Rules of the Workshop
• Share your ideas, experience, and questions with others• Respect different views and opinions
• Support a pleasantly learning environment
![Page 3: Theory-Driven Evaluation for Assessing and Improving · PDF file · 2011-02-20Theory-Driven Evaluation for Assessing and Improving ... and professors • Interventions ... Education/signage](https://reader031.fdocuments.net/reader031/viewer/2022022500/5aa4a7c67f8b9a7c1a8c6b3a/html5/thumbnails/3.jpg)
Part I: Basic Concepts and Conceptual Framework
![Page 4: Theory-Driven Evaluation for Assessing and Improving · PDF file · 2011-02-20Theory-Driven Evaluation for Assessing and Improving ... and professors • Interventions ... Education/signage](https://reader031.fdocuments.net/reader031/viewer/2022022500/5aa4a7c67f8b9a7c1a8c6b3a/html5/thumbnails/4.jpg)
Historical Background
An Alternative to method-driven and black-box evaluation :
• Method-Driven Evaluation
• Black-Box EvaluationIntervention Outcomes
• Theory-Driven Evaluation How? Why?
![Page 5: Theory-Driven Evaluation for Assessing and Improving · PDF file · 2011-02-20Theory-Driven Evaluation for Assessing and Improving ... and professors • Interventions ... Education/signage](https://reader031.fdocuments.net/reader031/viewer/2022022500/5aa4a7c67f8b9a7c1a8c6b3a/html5/thumbnails/5.jpg)
Literatures on Program Theory and Theory-Driven Evaluations
• New Directions of EvaluationBickman (Ed.), 1987, 1990Roger, et al., (Eds.) 2000
• Books Major evaluation text books (chapters on program theory)Chen, 1990Chen and Rossi, (eds.) 1992Chen, 2005Donaldson 2007
• Major evaluation text books have a chapter on theory- driven evaluation
• Many articles
![Page 6: Theory-Driven Evaluation for Assessing and Improving · PDF file · 2011-02-20Theory-Driven Evaluation for Assessing and Improving ... and professors • Interventions ... Education/signage](https://reader031.fdocuments.net/reader031/viewer/2022022500/5aa4a7c67f8b9a7c1a8c6b3a/html5/thumbnails/6.jpg)
Program Theory (1)
• Bickman (1987): a model of how a program is supposed to work.
• Wholey (1987): identifies program resources, program activities, intended program outcomes, and specifies a chain of causal assumptions linking these components.
• Weiss (1995): a picture of how and why programs work.
![Page 7: Theory-Driven Evaluation for Assessing and Improving · PDF file · 2011-02-20Theory-Driven Evaluation for Assessing and Improving ... and professors • Interventions ... Education/signage](https://reader031.fdocuments.net/reader031/viewer/2022022500/5aa4a7c67f8b9a7c1a8c6b3a/html5/thumbnails/7.jpg)
Definitions of program theory (2)
Chen (1990): What must be done to achieve the desirable goals, what other important impacts mayalso be anticipated, and how these goals and impacts would be generated.
Chen (2005): A set of stakeholders’ Implicit and explicit assumptions on what actions are required to solve a problem and why the problem will respond tothe actions.
Descriptive assumptions (Change Model)Prescriptive assumptions (Action Model )
![Page 8: Theory-Driven Evaluation for Assessing and Improving · PDF file · 2011-02-20Theory-Driven Evaluation for Assessing and Improving ... and professors • Interventions ... Education/signage](https://reader031.fdocuments.net/reader031/viewer/2022022500/5aa4a7c67f8b9a7c1a8c6b3a/html5/thumbnails/8.jpg)
Do stakeholders have a program theory underlying their program?
Do they have a change model? (Why would the intervention affect the outcome?)
Academic theory (explicit)Stakeholder theory (implicit)
Do they have an action model (Whatactions are needed?)
Who should be the implementers?How to recruit clients?How to deliver the intervention?etc.
![Page 9: Theory-Driven Evaluation for Assessing and Improving · PDF file · 2011-02-20Theory-Driven Evaluation for Assessing and Improving ... and professors • Interventions ... Education/signage](https://reader031.fdocuments.net/reader031/viewer/2022022500/5aa4a7c67f8b9a7c1a8c6b3a/html5/thumbnails/9.jpg)
Example of Change Model
• Stakeholder theory of a HIV Prevention Program for youth
Inviting HIV+ ? Safe speakers Sex
Condom SafeDistribution ? Sex
![Page 10: Theory-Driven Evaluation for Assessing and Improving · PDF file · 2011-02-20Theory-Driven Evaluation for Assessing and Improving ... and professors • Interventions ... Education/signage](https://reader031.fdocuments.net/reader031/viewer/2022022500/5aa4a7c67f8b9a7c1a8c6b3a/html5/thumbnails/10.jpg)
Examples of Stakeholder Theory : Laub et al., 1999
• Stakeholder theory of a HIV Prevention Program for youth
Inviting HIV+ Youth feel they are Safe speakers not invincible for HIV Sex
Condom Condom SafeDistribution Availability Sex
![Page 11: Theory-Driven Evaluation for Assessing and Improving · PDF file · 2011-02-20Theory-Driven Evaluation for Assessing and Improving ... and professors • Interventions ... Education/signage](https://reader031.fdocuments.net/reader031/viewer/2022022500/5aa4a7c67f8b9a7c1a8c6b3a/html5/thumbnails/11.jpg)
PROGRAM THEORY
Implementing
organizations
Implementers
Intervention Determinants Outcomes
Associate organizations and
community partners
Ecological context
Intervention and
service delivery protocols
Target populations
Change Model
Action Model
![Page 12: Theory-Driven Evaluation for Assessing and Improving · PDF file · 2011-02-20Theory-Driven Evaluation for Assessing and Improving ... and professors • Interventions ... Education/signage](https://reader031.fdocuments.net/reader031/viewer/2022022500/5aa4a7c67f8b9a7c1a8c6b3a/html5/thumbnails/12.jpg)
Theory-Driven Evaluation
• Evaluators facilitate program stakeholders to clarify their program theory. (program theory: stakeholders’ implicit and explicit assumptions on what actions are required to solve asocial or health problem and how the problem willrespond to the actions.)
• The program theory is then used as a framework toguide the design of evaluation design, the selection
of research methods, and the collection of data.
![Page 13: Theory-Driven Evaluation for Assessing and Improving · PDF file · 2011-02-20Theory-Driven Evaluation for Assessing and Improving ... and professors • Interventions ... Education/signage](https://reader031.fdocuments.net/reader031/viewer/2022022500/5aa4a7c67f8b9a7c1a8c6b3a/html5/thumbnails/13.jpg)
Applications of Theory-Driven Evaluation
• Theory-driven outcome evaluation (change model)
• Theory-driven process evaluation (action model)
• Theory-driven approach for program planning
(action model and change model)
![Page 14: Theory-Driven Evaluation for Assessing and Improving · PDF file · 2011-02-20Theory-Driven Evaluation for Assessing and Improving ... and professors • Interventions ... Education/signage](https://reader031.fdocuments.net/reader031/viewer/2022022500/5aa4a7c67f8b9a7c1a8c6b3a/html5/thumbnails/14.jpg)
Part II: Theory-Driven Outcome Evaluation
![Page 15: Theory-Driven Evaluation for Assessing and Improving · PDF file · 2011-02-20Theory-Driven Evaluation for Assessing and Improving ... and professors • Interventions ... Education/signage](https://reader031.fdocuments.net/reader031/viewer/2022022500/5aa4a7c67f8b9a7c1a8c6b3a/html5/thumbnails/15.jpg)
Change Model
Intervention Determinants Outcomes
![Page 16: Theory-Driven Evaluation for Assessing and Improving · PDF file · 2011-02-20Theory-Driven Evaluation for Assessing and Improving ... and professors • Interventions ... Education/signage](https://reader031.fdocuments.net/reader031/viewer/2022022500/5aa4a7c67f8b9a7c1a8c6b3a/html5/thumbnails/16.jpg)
Components of a Change Model
– Intervention
– Determinants: Causes of a ProblemLeverages
Intervening variablesMediators
– Outcomes
![Page 17: Theory-Driven Evaluation for Assessing and Improving · PDF file · 2011-02-20Theory-Driven Evaluation for Assessing and Improving ... and professors • Interventions ... Education/signage](https://reader031.fdocuments.net/reader031/viewer/2022022500/5aa4a7c67f8b9a7c1a8c6b3a/html5/thumbnails/17.jpg)
Examples of Stakeholder Theory : Laub et al., 1999
• Stakeholder theory of a HIV Prevention Program for youth
Inviting HIV+ Youth feel they are Safe speakers not invincible for HIV Sex
Condom Condom SafeDistribution Availability Sex
![Page 18: Theory-Driven Evaluation for Assessing and Improving · PDF file · 2011-02-20Theory-Driven Evaluation for Assessing and Improving ... and professors • Interventions ... Education/signage](https://reader031.fdocuments.net/reader031/viewer/2022022500/5aa4a7c67f8b9a7c1a8c6b3a/html5/thumbnails/18.jpg)
Academic theory based intervention: Stretcher, et al., 1989
Home-based intervention based upon
social learning theory
Outcome expectancy: Mothers’ perception of the effect of passive
smoking
Efficacy expectation: Mothers’ perception of
their ability to maintain a smoking-free environment
Reducing infants’Passive smoking
![Page 19: Theory-Driven Evaluation for Assessing and Improving · PDF file · 2011-02-20Theory-Driven Evaluation for Assessing and Improving ... and professors • Interventions ... Education/signage](https://reader031.fdocuments.net/reader031/viewer/2022022500/5aa4a7c67f8b9a7c1a8c6b3a/html5/thumbnails/19.jpg)
Example of Change models
PreschoolProgram for
4 yr. oldchildren
Lean appropriateSchool behavior
Improved pre-Literacy skills
Positive attitudesToward school
Increase schoolreadiness
Greater cognitive gains inkindergarten
![Page 20: Theory-Driven Evaluation for Assessing and Improving · PDF file · 2011-02-20Theory-Driven Evaluation for Assessing and Improving ... and professors • Interventions ... Education/signage](https://reader031.fdocuments.net/reader031/viewer/2022022500/5aa4a7c67f8b9a7c1a8c6b3a/html5/thumbnails/20.jpg)
Exercise 1: Academic vs. Stakeholder Theory
• Program goal: Reduce exposure to secondhand smoke among residents of low-income housing project
• Planning group: Stakeholders (NGOs) and professors • Interventions:
Professors: Community health advisor (CHA) modelStakeholders: Education/signage intervention (Encourage residents to place signage such as window decal: “ This is a Smoke-Free Home.”)
• Outreach workers: 7 female residents receiving unemployment checks
• Budget: moderate
Exercise questions: 1. Discuss the rationales used by stakeholders and professors for
proposing different interventions 2. Discuss the pros and cons of these two types of interventions
![Page 21: Theory-Driven Evaluation for Assessing and Improving · PDF file · 2011-02-20Theory-Driven Evaluation for Assessing and Improving ... and professors • Interventions ... Education/signage](https://reader031.fdocuments.net/reader031/viewer/2022022500/5aa4a7c67f8b9a7c1a8c6b3a/html5/thumbnails/21.jpg)
Program theory underlying the intervention
Distributinganti-smoking
kit +
encouraging of placing window decal
CHA counseling
Signage placed in window,
table, etc.
Self-efficacy
Reducing exposure
of secondhand smoke
![Page 22: Theory-Driven Evaluation for Assessing and Improving · PDF file · 2011-02-20Theory-Driven Evaluation for Assessing and Improving ... and professors • Interventions ... Education/signage](https://reader031.fdocuments.net/reader031/viewer/2022022500/5aa4a7c67f8b9a7c1a8c6b3a/html5/thumbnails/22.jpg)
Conceptual Facilitation
• Evaluators’ role: facilitator• Principles of facilitation:
RespectFairness Parsimony Stakeholders’ ownership
![Page 23: Theory-Driven Evaluation for Assessing and Improving · PDF file · 2011-02-20Theory-Driven Evaluation for Assessing and Improving ... and professors • Interventions ... Education/signage](https://reader031.fdocuments.net/reader031/viewer/2022022500/5aa4a7c67f8b9a7c1a8c6b3a/html5/thumbnails/23.jpg)
How to clarify stakeholders’ change model or program theory (conceptualization facilitation)
• Facilitating stakeholders to clarify their program theory
• Formats– Intensive interview– Working group meeting
• Theorizing Methods– Forward reasoning– Backward reasoning– Both
![Page 24: Theory-Driven Evaluation for Assessing and Improving · PDF file · 2011-02-20Theory-Driven Evaluation for Assessing and Improving ... and professors • Interventions ... Education/signage](https://reader031.fdocuments.net/reader031/viewer/2022022500/5aa4a7c67f8b9a7c1a8c6b3a/html5/thumbnails/24.jpg)
Clarifying Stakeholders’ Change Model:
1. Clarifying Goals/Outcomes
• Avoid a goal trapOfficial goals vs. operative goals Ex. Official goals of a prison (rehabilitation?)
• Stress the measurabilityEnhancing elderly people’s social functioning
• Stress the plausibility of goalsDesirable goals vs. plausible goals Ex. The goal of a media program is to
eliminate racism
• Issues on intended and unintended outcomes
Unintended outcomes: negativepositive
![Page 25: Theory-Driven Evaluation for Assessing and Improving · PDF file · 2011-02-20Theory-Driven Evaluation for Assessing and Improving ... and professors • Interventions ... Education/signage](https://reader031.fdocuments.net/reader031/viewer/2022022500/5aa4a7c67f8b9a7c1a8c6b3a/html5/thumbnails/25.jpg)
2. Clarifying Determinants
– Determinants: Causes of a problem, leverages,intervening variables, mediators
– Stakeholders usually make a set of implicit assumptions on determinants when they design an intervention program
Ex. Wife abuse programWhy do husbands abuse their wife?power factorscultural factors (e.g. rule of thumb)anger control criminal justice factors
![Page 26: Theory-Driven Evaluation for Assessing and Improving · PDF file · 2011-02-20Theory-Driven Evaluation for Assessing and Improving ... and professors • Interventions ... Education/signage](https://reader031.fdocuments.net/reader031/viewer/2022022500/5aa4a7c67f8b9a7c1a8c6b3a/html5/thumbnails/26.jpg)
Stakeholders’ change model
Group therapy
Skills for controlling anger and frustration
Knowledge on the legal aspect of
abuse
Reducing abusive behaviors
![Page 27: Theory-Driven Evaluation for Assessing and Improving · PDF file · 2011-02-20Theory-Driven Evaluation for Assessing and Improving ... and professors • Interventions ... Education/signage](https://reader031.fdocuments.net/reader031/viewer/2022022500/5aa4a7c67f8b9a7c1a8c6b3a/html5/thumbnails/27.jpg)
3. Clarifying Intervention
• What is the intervention? What are the essential elements?
Intervention vs. supportive activities?
• Do stakeholder groups agree on the intervention?
![Page 28: Theory-Driven Evaluation for Assessing and Improving · PDF file · 2011-02-20Theory-Driven Evaluation for Assessing and Improving ... and professors • Interventions ... Education/signage](https://reader031.fdocuments.net/reader031/viewer/2022022500/5aa4a7c67f8b9a7c1a8c6b3a/html5/thumbnails/28.jpg)
Conceptualization Facilitation
Theorizing Methods– Backward reasoning
Why? Intervention Determinants Outcomes
– Forward reasoningIntervention Why? Outcomes
(Determinants)– Both
![Page 29: Theory-Driven Evaluation for Assessing and Improving · PDF file · 2011-02-20Theory-Driven Evaluation for Assessing and Improving ... and professors • Interventions ... Education/signage](https://reader031.fdocuments.net/reader031/viewer/2022022500/5aa4a7c67f8b9a7c1a8c6b3a/html5/thumbnails/29.jpg)
Use a change model to conduct outcome evaluation
• Clarifying program theory Qualitative methods
• MeasuresFind indicators or develop instruments for measuring intervention, determinants, and outcomes
• Research designUsing a rigorous design (i.e., experimental and quasi-experimental design) to provide credible evidence among these components
• Data collectionQuantitative methods
• Statistical AnalysisPath analysis, structural equations model
![Page 30: Theory-Driven Evaluation for Assessing and Improving · PDF file · 2011-02-20Theory-Driven Evaluation for Assessing and Improving ... and professors • Interventions ... Education/signage](https://reader031.fdocuments.net/reader031/viewer/2022022500/5aa4a7c67f8b9a7c1a8c6b3a/html5/thumbnails/30.jpg)
Advantages to evaluate a change model
1. Understanding why a program was successful or not
• Action theory: Success or fail
• Conceptual theory: Success or fail
![Page 31: Theory-Driven Evaluation for Assessing and Improving · PDF file · 2011-02-20Theory-Driven Evaluation for Assessing and Improving ... and professors • Interventions ... Education/signage](https://reader031.fdocuments.net/reader031/viewer/2022022500/5aa4a7c67f8b9a7c1a8c6b3a/html5/thumbnails/31.jpg)
Concepts of Program Success or Failure
Intervention Determinants Outcomes
Action TheorySuccess or failure
Conceptual theory Success or failure
Condom vending Machines in schools
Condom availability Increase safesex
![Page 32: Theory-Driven Evaluation for Assessing and Improving · PDF file · 2011-02-20Theory-Driven Evaluation for Assessing and Improving ... and professors • Interventions ... Education/signage](https://reader031.fdocuments.net/reader031/viewer/2022022500/5aa4a7c67f8b9a7c1a8c6b3a/html5/thumbnails/32.jpg)
2. Enhance construct validity of evaluation
Family planningprogram
Knowledge, skills, andvalues on birth control
Reducing fertility rate of
a region
Other channels?
![Page 33: Theory-Driven Evaluation for Assessing and Improving · PDF file · 2011-02-20Theory-Driven Evaluation for Assessing and Improving ... and professors • Interventions ... Education/signage](https://reader031.fdocuments.net/reader031/viewer/2022022500/5aa4a7c67f8b9a7c1a8c6b3a/html5/thumbnails/33.jpg)
3. Formative feedback on the mediating process for early improvement
Protein Supplementation
Malnutrition
Physical &Psychological
growth
Action theoryConceptual theory
![Page 34: Theory-Driven Evaluation for Assessing and Improving · PDF file · 2011-02-20Theory-Driven Evaluation for Assessing and Improving ... and professors • Interventions ... Education/signage](https://reader031.fdocuments.net/reader031/viewer/2022022500/5aa4a7c67f8b9a7c1a8c6b3a/html5/thumbnails/34.jpg)
Methods Used to Assess a Change Model
Change Model Empirical assessmentClarification ______________________Qualitative methods Quantitative methods :
In-depth interview Design : Working group Experimental, quasi-
experimental, pre-experimental designsStatistical Model :Path analysisStructural equation model
Qualitative Methods: optional
![Page 35: Theory-Driven Evaluation for Assessing and Improving · PDF file · 2011-02-20Theory-Driven Evaluation for Assessing and Improving ... and professors • Interventions ... Education/signage](https://reader031.fdocuments.net/reader031/viewer/2022022500/5aa4a7c67f8b9a7c1a8c6b3a/html5/thumbnails/35.jpg)
Part II: Theory-Driven Approach for Program Planning
![Page 36: Theory-Driven Evaluation for Assessing and Improving · PDF file · 2011-02-20Theory-Driven Evaluation for Assessing and Improving ... and professors • Interventions ... Education/signage](https://reader031.fdocuments.net/reader031/viewer/2022022500/5aa4a7c67f8b9a7c1a8c6b3a/html5/thumbnails/36.jpg)
Program Theory and Program Planning
• Facilitating stakeholders to clarify their program theory • Stakeholders want evaluators to help in the planning
stage – To enhance the soundness of program theory– To build a consensus on the program theory among
different stakeholder groups before implementation
![Page 37: Theory-Driven Evaluation for Assessing and Improving · PDF file · 2011-02-20Theory-Driven Evaluation for Assessing and Improving ... and professors • Interventions ... Education/signage](https://reader031.fdocuments.net/reader031/viewer/2022022500/5aa4a7c67f8b9a7c1a8c6b3a/html5/thumbnails/37.jpg)
Consensus issue and the implication of evaluation criteria
Ex. The first Head Start evaluation
• Evaluators mainly assessed the program by using the following goals proposed by federal government: math and reading scores
The evaluation results: the program has little effects on math and reading score
• Managers and teachers of the local head start centers: The evaluators evaluate wrong goals.
Their goals: nutrition, physical and emotional development, dental hygiene,social skills, and parenthood.
![Page 38: Theory-Driven Evaluation for Assessing and Improving · PDF file · 2011-02-20Theory-Driven Evaluation for Assessing and Improving ... and professors • Interventions ... Education/signage](https://reader031.fdocuments.net/reader031/viewer/2022022500/5aa4a7c67f8b9a7c1a8c6b3a/html5/thumbnails/38.jpg)
Part III: Theory-Driven Process Evaluation
![Page 39: Theory-Driven Evaluation for Assessing and Improving · PDF file · 2011-02-20Theory-Driven Evaluation for Assessing and Improving ... and professors • Interventions ... Education/signage](https://reader031.fdocuments.net/reader031/viewer/2022022500/5aa4a7c67f8b9a7c1a8c6b3a/html5/thumbnails/39.jpg)
Action model as Planned
Action model as delivered
Congruency
![Page 40: Theory-Driven Evaluation for Assessing and Improving · PDF file · 2011-02-20Theory-Driven Evaluation for Assessing and Improving ... and professors • Interventions ... Education/signage](https://reader031.fdocuments.net/reader031/viewer/2022022500/5aa4a7c67f8b9a7c1a8c6b3a/html5/thumbnails/40.jpg)
Theory-Driven Process Evaluation
• Clarifying stakeholders’ action model • Applying conceptualization facilitation
Intensive interview or working group meeting • Research methods used to collecting data for assessing
the actual implementation: mixed methods• Assessing the congruency between the plan and actual
implementation
![Page 41: Theory-Driven Evaluation for Assessing and Improving · PDF file · 2011-02-20Theory-Driven Evaluation for Assessing and Improving ... and professors • Interventions ... Education/signage](https://reader031.fdocuments.net/reader031/viewer/2022022500/5aa4a7c67f8b9a7c1a8c6b3a/html5/thumbnails/41.jpg)
Issues on Incongruence between plan vs. actual implementation
• Fidelity tradition• Adaptation tradition
![Page 42: Theory-Driven Evaluation for Assessing and Improving · PDF file · 2011-02-20Theory-Driven Evaluation for Assessing and Improving ... and professors • Interventions ... Education/signage](https://reader031.fdocuments.net/reader031/viewer/2022022500/5aa4a7c67f8b9a7c1a8c6b3a/html5/thumbnails/42.jpg)
Issue 2: Theory-Driven Evaluations and Mixed Methods
Theory ClarificationQualitative
Quantitative
Empirical AssessmentQuantitative (Switch)Mixed methods for different
components (Complementarity)
Mixed methods for including contextual information
(Contextual overlaying) Mixed methods for
triangulation (Triangulation)
Qualitative (Switch)Mixed methods, etc.
![Page 43: Theory-Driven Evaluation for Assessing and Improving · PDF file · 2011-02-20Theory-Driven Evaluation for Assessing and Improving ... and professors • Interventions ... Education/signage](https://reader031.fdocuments.net/reader031/viewer/2022022500/5aa4a7c67f8b9a7c1a8c6b3a/html5/thumbnails/43.jpg)
Strategies for Conceptualization Facilitation
• Same as those in the change model
![Page 44: Theory-Driven Evaluation for Assessing and Improving · PDF file · 2011-02-20Theory-Driven Evaluation for Assessing and Improving ... and professors • Interventions ... Education/signage](https://reader031.fdocuments.net/reader031/viewer/2022022500/5aa4a7c67f8b9a7c1a8c6b3a/html5/thumbnails/44.jpg)
Facilitating Stakeholders to Clarify their Action M odel
• Implementing organizations : Assess, enhance, and ensure its capacities
• Implementers : recruit, train, and maintain both competency and commitment
• Intervention protocol : Make it available • Associate organizations : Establish collaboration• Ecological context : seek its support• Target population : identify, recruit, screen, serve• Goals/Outcomes: measurability, plausibility
![Page 45: Theory-Driven Evaluation for Assessing and Improving · PDF file · 2011-02-20Theory-Driven Evaluation for Assessing and Improving ... and professors • Interventions ... Education/signage](https://reader031.fdocuments.net/reader031/viewer/2022022500/5aa4a7c67f8b9a7c1a8c6b3a/html5/thumbnails/45.jpg)
Theory-Driven Process Evaluation
Program componentsin an action model Program Plan Actual implementation
Target population
Implementing org.
Implementers
Intervention and servicedelivery protocols
Associated orgs./partners
Ecological support
Data (quat.Or qual.)
![Page 46: Theory-Driven Evaluation for Assessing and Improving · PDF file · 2011-02-20Theory-Driven Evaluation for Assessing and Improving ... and professors • Interventions ... Education/signage](https://reader031.fdocuments.net/reader031/viewer/2022022500/5aa4a7c67f8b9a7c1a8c6b3a/html5/thumbnails/46.jpg)
Theory-Driven Process Evaluation in Action: Evaluating a School-Based Anti-Drug Abuse Program
in Taiwan
• Drug abuse among middle school students had worsened
• The Ministry of Education launched a national anti-drug abuse program to deal with the problem
• Teachers were trained to identify drug abusing students and provide counseling
• Schools were required to file monthly reports on the numbers of active drug abusing students to the ministry
![Page 47: Theory-Driven Evaluation for Assessing and Improving · PDF file · 2011-02-20Theory-Driven Evaluation for Assessing and Improving ... and professors • Interventions ... Education/signage](https://reader031.fdocuments.net/reader031/viewer/2022022500/5aa4a7c67f8b9a7c1a8c6b3a/html5/thumbnails/47.jpg)
# of activecases
1 2 3 yr
*3850
*1625
*260*55
*501*353
*440*374
![Page 48: Theory-Driven Evaluation for Assessing and Improving · PDF file · 2011-02-20Theory-Driven Evaluation for Assessing and Improving ... and professors • Interventions ... Education/signage](https://reader031.fdocuments.net/reader031/viewer/2022022500/5aa4a7c67f8b9a7c1a8c6b3a/html5/thumbnails/48.jpg)
Theory-Driven Process Evaluation Application Procedures
• Working group meetings with key officials at the Ministry of Education to develop an action model
• Working group meetings with representatives of teachers to develop their version of the action model
• Synthesized two groups into a combined version for feedback
• Used mixed methods (site visits, survey, participant observation, focus group meetings, interviews, record checking) to collect implementation data
![Page 49: Theory-Driven Evaluation for Assessing and Improving · PDF file · 2011-02-20Theory-Driven Evaluation for Assessing and Improving ... and professors • Interventions ... Education/signage](https://reader031.fdocuments.net/reader031/viewer/2022022500/5aa4a7c67f8b9a7c1a8c6b3a/html5/thumbnails/49.jpg)
Action Model as Planned vs. as Implemented
Component Plan Actual implementation
Target populationAll drug abusing
studentsVerified through
urinalysis
Easy to reach students
Urinalysis was not controlled
Implementers Competent in delivering the intervention
Inadequately trained
![Page 50: Theory-Driven Evaluation for Assessing and Improving · PDF file · 2011-02-20Theory-Driven Evaluation for Assessing and Improving ... and professors • Interventions ... Education/signage](https://reader031.fdocuments.net/reader031/viewer/2022022500/5aa4a7c67f8b9a7c1a8c6b3a/html5/thumbnails/50.jpg)
Action Model (cont)
Component Plan Actual implementation
Intervention protocol
High quality counseling
Admonishments, threats,
encouragements
Service delivery protocol
Compulsory individual
counseling
Lacked plan and objectives
![Page 51: Theory-Driven Evaluation for Assessing and Improving · PDF file · 2011-02-20Theory-Driven Evaluation for Assessing and Improving ... and professors • Interventions ... Education/signage](https://reader031.fdocuments.net/reader031/viewer/2022022500/5aa4a7c67f8b9a7c1a8c6b3a/html5/thumbnails/51.jpg)
Action Model (cont)
Component Plan Actual implementation
Implementing Organizations
Every School Smaller schools not involved
Linking with associate
organizations
Effective centralized school
system
Communication gap; mistrust
between schools and ministry of
education
![Page 52: Theory-Driven Evaluation for Assessing and Improving · PDF file · 2011-02-20Theory-Driven Evaluation for Assessing and Improving ... and professors • Interventions ... Education/signage](https://reader031.fdocuments.net/reader031/viewer/2022022500/5aa4a7c67f8b9a7c1a8c6b3a/html5/thumbnails/52.jpg)
Action Model (cont)
Component Plan Actual implementation
Ecological Context
Micro Eliminating video game arcades
Video game arcades still exist
Macro Strong public support
Strong public support
![Page 53: Theory-Driven Evaluation for Assessing and Improving · PDF file · 2011-02-20Theory-Driven Evaluation for Assessing and Improving ... and professors • Interventions ... Education/signage](https://reader031.fdocuments.net/reader031/viewer/2022022500/5aa4a7c67f8b9a7c1a8c6b3a/html5/thumbnails/53.jpg)
Part IV: Advanced Issues
• Top-down approach and evidence-based intervention
![Page 54: Theory-Driven Evaluation for Assessing and Improving · PDF file · 2011-02-20Theory-Driven Evaluation for Assessing and Improving ... and professors • Interventions ... Education/signage](https://reader031.fdocuments.net/reader031/viewer/2022022500/5aa4a7c67f8b9a7c1a8c6b3a/html5/thumbnails/54.jpg)
Evidence –Based Interventions (EBIs) and the Top-Down Approach
• EBIs: Interventions proven efficacious by rigorous methods in controlled settings. Rigorous methods usually means randomized controlled trials (RCTs).
• The top-down approach:1. Efficacy evaluation (EBIs): Providing strongest
evidence of effectuality of an intervention (Maximizing internal validity)
2. Effectiveness evaluation: Providing evidence that the effectuality is transferable to the real-world
(external validity) 3. Dissemination
![Page 55: Theory-Driven Evaluation for Assessing and Improving · PDF file · 2011-02-20Theory-Driven Evaluation for Assessing and Improving ... and professors • Interventions ... Education/signage](https://reader031.fdocuments.net/reader031/viewer/2022022500/5aa4a7c67f8b9a7c1a8c6b3a/html5/thumbnails/55.jpg)
Limitations of the EBIs:
1. 1. EBIs are not necessarily to be effective in the real world.
2. EBIs are not relevant to real-world operations.
3. EBIs do not adequately address issues and interests of stakeholders.
4. EBIs Can be implemented by stakeholders with high fidelity in the real-world context.
![Page 56: Theory-Driven Evaluation for Assessing and Improving · PDF file · 2011-02-20Theory-Driven Evaluation for Assessing and Improving ... and professors • Interventions ... Education/signage](https://reader031.fdocuments.net/reader031/viewer/2022022500/5aa4a7c67f8b9a7c1a8c6b3a/html5/thumbnails/56.jpg)
Limitation #4: Difficulties in implementing EBIs in the real world
• National Cooperative Inner-City Asthma Study (NCICAS): Trained master’s level social workers to provide families asthma and psychosocial counseling.
• NCICAS had features of an efficacy evaluation such as: -- monetary and child care incentives--highly committed counselors, --food/refreshments during counseling, --frequent contacts with participants, --counseling sessions were held at regular hours, etc.
![Page 57: Theory-Driven Evaluation for Assessing and Improving · PDF file · 2011-02-20Theory-Driven Evaluation for Assessing and Improving ... and professors • Interventions ... Education/signage](https://reader031.fdocuments.net/reader031/viewer/2022022500/5aa4a7c67f8b9a7c1a8c6b3a/html5/thumbnails/57.jpg)
Limitation #4: Difficulties in implementing EBI in the real world (continued)
• The Inner-City Asthma Intervention (ICAI): Implementing NCICA as an intervention in the real-world.
• Difficulties in delivering the exact NCICAS in the real world: Many adaptations and changes.
-Were difficulties to contact and meet with families -Held sessions in evenings or weekends-Provided no monetary and child care incentives-Provided no food/refreshments- Had difficulties in retaining social workers
• Only 25% of the children completed the intervention
![Page 58: Theory-Driven Evaluation for Assessing and Improving · PDF file · 2011-02-20Theory-Driven Evaluation for Assessing and Improving ... and professors • Interventions ... Education/signage](https://reader031.fdocuments.net/reader031/viewer/2022022500/5aa4a7c67f8b9a7c1a8c6b3a/html5/thumbnails/58.jpg)
PROGRAM THEORY
Implementing
organizations
Implementers
Intervention Determinants Outcomes
Associate organizations and
community partners
Ecological context
Intervention and
service delivery protocols
Target populations
Change Model
Action Model
![Page 59: Theory-Driven Evaluation for Assessing and Improving · PDF file · 2011-02-20Theory-Driven Evaluation for Assessing and Improving ... and professors • Interventions ... Education/signage](https://reader031.fdocuments.net/reader031/viewer/2022022500/5aa4a7c67f8b9a7c1a8c6b3a/html5/thumbnails/59.jpg)
Integrative Validity Model
• Effectual validity : Evidence on an intervention’s effectuality
• Viable validity: Evidence on an intervention’s viability
• Transferable validity: Evidence on transferability of an intervention’s effectiveness and /or viability
* The model is an expansion of the distinction of internal and external validity by Campbell and Stanley’s (1963)
![Page 60: Theory-Driven Evaluation for Assessing and Improving · PDF file · 2011-02-20Theory-Driven Evaluation for Assessing and Improving ... and professors • Interventions ... Education/signage](https://reader031.fdocuments.net/reader031/viewer/2022022500/5aa4a7c67f8b9a7c1a8c6b3a/html5/thumbnails/60.jpg)
An Alternative: Integrative Validity Model and “Bot tom -Up”
Approach (Continued )
Concept of Viability Components:
Practical, suitable, affordable, evaluable, and helpful.
![Page 61: Theory-Driven Evaluation for Assessing and Improving · PDF file · 2011-02-20Theory-Driven Evaluation for Assessing and Improving ... and professors • Interventions ... Education/signage](https://reader031.fdocuments.net/reader031/viewer/2022022500/5aa4a7c67f8b9a7c1a8c6b3a/html5/thumbnails/61.jpg)
An Alternative: Integrative Validity Model and “Bot tom -Up”
Approach (Continued )
Viability Evaluation • Assess the extent to which an intervention program
is viable in the real world (e.g., practical, suitable, affordable, evaluable, helpful)
• Methodology: Mixed methods (e.g., pretest-posttest, interviews, focus groups, survey)
![Page 62: Theory-Driven Evaluation for Assessing and Improving · PDF file · 2011-02-20Theory-Driven Evaluation for Assessing and Improving ... and professors • Interventions ... Education/signage](https://reader031.fdocuments.net/reader031/viewer/2022022500/5aa4a7c67f8b9a7c1a8c6b3a/html5/thumbnails/62.jpg)
The Bottom -Up Approach
– Start with addressing viable validity (viable evaluation), optimize effectual validity and transferable validity (effectiveness evaluation),then maximize effectual validity (efficacy evaluation)
– Only viable interventions are worthy of effectiveness evaluation
– Only those interventions that are viable, effective, and capable of generalization are worthy of efficacy evaluation
![Page 63: Theory-Driven Evaluation for Assessing and Improving · PDF file · 2011-02-20Theory-Driven Evaluation for Assessing and Improving ... and professors • Interventions ... Education/signage](https://reader031.fdocuments.net/reader031/viewer/2022022500/5aa4a7c67f8b9a7c1a8c6b3a/html5/thumbnails/63.jpg)
Efficacy Evaluation
Effectiveness Evaluation
Dissemination
Viability Evaluation
Effectiveness Evaluation
Efficacy Evaluation
Dissemination
Top-Down Approach
Bottom-Up Approach
![Page 64: Theory-Driven Evaluation for Assessing and Improving · PDF file · 2011-02-20Theory-Driven Evaluation for Assessing and Improving ... and professors • Interventions ... Education/signage](https://reader031.fdocuments.net/reader031/viewer/2022022500/5aa4a7c67f8b9a7c1a8c6b3a/html5/thumbnails/64.jpg)
Exercise 2: What are the pros and cons of the top-down approach and bottom -up approach ?
• Top-down approachProsCons
• Bottom-up approach Pros Cons
![Page 65: Theory-Driven Evaluation for Assessing and Improving · PDF file · 2011-02-20Theory-Driven Evaluation for Assessing and Improving ... and professors • Interventions ... Education/signage](https://reader031.fdocuments.net/reader031/viewer/2022022500/5aa4a7c67f8b9a7c1a8c6b3a/html5/thumbnails/65.jpg)
References
• Chen, H.T. 2005. Practical Program Evaluation: assessing and improving program planning, implementation, and effectiveness. Sage.
• Chen, H.T. 1990. Theory-Driven Evaluations. Sage.
![Page 66: Theory-Driven Evaluation for Assessing and Improving · PDF file · 2011-02-20Theory-Driven Evaluation for Assessing and Improving ... and professors • Interventions ... Education/signage](https://reader031.fdocuments.net/reader031/viewer/2022022500/5aa4a7c67f8b9a7c1a8c6b3a/html5/thumbnails/66.jpg)
References• Chen HT. 2010. “The bottom-up approach to integrative
validity: a new perspective for program evaluation.”Evaluation and Program Planning 33(3): 205–14.
• Chen HT, Donaldson S, Mark M, (Eds.) Advancing Validity in Outcome Evaluation: Theory and Practice.New Directions for Evaluation (forthcoming).
• Chen HT and Garbe, P, “Assessing program outcomes from the bottom-up approach: An innovative perspective to outcome evaluation”, in Chen HT, Donaldson S, Mark M, (Eds.) Advancing Validity in Outcome Evaluation: Theory and Practice. New Directions for Evaluation (forthcoming).