Health Policy Evaluation - Ministry of Public...
Transcript of Health Policy Evaluation - Ministry of Public...
![Page 1: Health Policy Evaluation - Ministry of Public Healthbps.moph.go.th/new_bps/sites/default/files/Health... · Program evaluation A systematic method for collecting, analyzing, and using](https://reader033.fdocuments.net/reader033/viewer/2022053004/5f086d377e708231d421f48c/html5/thumbnails/1.jpg)
Health Policy Evaluation:Concept and Application
PITAKPOL BOONYALIK, MD, PHD, MBA, MSC
![Page 2: Health Policy Evaluation - Ministry of Public Healthbps.moph.go.th/new_bps/sites/default/files/Health... · Program evaluation A systematic method for collecting, analyzing, and using](https://reader033.fdocuments.net/reader033/viewer/2022053004/5f086d377e708231d421f48c/html5/thumbnails/2.jpg)
Health Policy Evaluation: Concept and Application
Program evaluation concept
Frameworks for program evaluation
Steps to do program evaluation
Case studies
2
![Page 3: Health Policy Evaluation - Ministry of Public Healthbps.moph.go.th/new_bps/sites/default/files/Health... · Program evaluation A systematic method for collecting, analyzing, and using](https://reader033.fdocuments.net/reader033/viewer/2022053004/5f086d377e708231d421f48c/html5/thumbnails/3.jpg)
Program evaluation
A systematic method for collecting, analyzing, and
using information to answer questions about projects,
policies and programs, particularly about their
effectiveness and efficiency. In both the public and
private sectors, stakeholders often want to know
whether the programs they are funding, implementing,
voting for, receiving or objecting to are producing the intended effect.
3
Source: Administration for Children and Families (2010) The Program Manager's Guide to Evaluation. Chapter 2: What is program evaluation? Retrieved from https://en.wikipedia.org/wiki/Program_evaluation#CIPP_Model_of_evaluation
![Page 4: Health Policy Evaluation - Ministry of Public Healthbps.moph.go.th/new_bps/sites/default/files/Health... · Program evaluation A systematic method for collecting, analyzing, and using](https://reader033.fdocuments.net/reader033/viewer/2022053004/5f086d377e708231d421f48c/html5/thumbnails/4.jpg)
Evaluation vs Research4
![Page 5: Health Policy Evaluation - Ministry of Public Healthbps.moph.go.th/new_bps/sites/default/files/Health... · Program evaluation A systematic method for collecting, analyzing, and using](https://reader033.fdocuments.net/reader033/viewer/2022053004/5f086d377e708231d421f48c/html5/thumbnails/5.jpg)
Evaluation : Systematic approach
Assessment of the need for the program
Assessment of program theory
Assessment of implementation
Assessment of the program's outcome or impact
Assessment of the program's cost and efficiency
5
Source: Rossi, P., Lipsey, M.W., & Freeman, H.E. (2004). Evaluation: a systematic approach (7th ed.). Thousand Oaks: Sage.
![Page 6: Health Policy Evaluation - Ministry of Public Healthbps.moph.go.th/new_bps/sites/default/files/Health... · Program evaluation A systematic method for collecting, analyzing, and using](https://reader033.fdocuments.net/reader033/viewer/2022053004/5f086d377e708231d421f48c/html5/thumbnails/6.jpg)
ปจจยแหงความส าเรจของโปรแกรมสขภาพ
การประเมนความตองการ (Need assessment)การออกแบบโปรแกรมโดยทฤษฎหรอหลกฐานเชง
ประจกษ (Theory-driven program and evidence-based program planning)
การมสวนรวมของชมชน (Community participation)
6
![Page 7: Health Policy Evaluation - Ministry of Public Healthbps.moph.go.th/new_bps/sites/default/files/Health... · Program evaluation A systematic method for collecting, analyzing, and using](https://reader033.fdocuments.net/reader033/viewer/2022053004/5f086d377e708231d421f48c/html5/thumbnails/7.jpg)
Assessing needs
the processes or methods used by evaluators to describe and diagnose social needs
7
Source: Rossi, P., Lipsey, M.W., & Freeman, H.E. (2004). Evaluation: a systematic approach (7th ed.). Thousand Oaks: Sage.
![Page 8: Health Policy Evaluation - Ministry of Public Healthbps.moph.go.th/new_bps/sites/default/files/Health... · Program evaluation A systematic method for collecting, analyzing, and using](https://reader033.fdocuments.net/reader033/viewer/2022053004/5f086d377e708231d421f48c/html5/thumbnails/8.jpg)
Assessing needs (cont.)
Define the problem
Assess the extent of the problem
Define and identify the target of interventions
8
Source: Rossi, P., Lipsey, M.W., & Freeman, H.E. (2004). Evaluation: a systematic approach (7th ed.). Thousand Oaks: Sage.
![Page 9: Health Policy Evaluation - Ministry of Public Healthbps.moph.go.th/new_bps/sites/default/files/Health... · Program evaluation A systematic method for collecting, analyzing, and using](https://reader033.fdocuments.net/reader033/viewer/2022053004/5f086d377e708231d421f48c/html5/thumbnails/9.jpg)
9
คณสมบตของเครองมอประเมนทางสขภาพ
ความเทยง (Reliability) = ความสามารถของเครองมอในการใหผลลพธเหมอนเดมInternal consistency reliability
เชน Cronbach’s alpha, split half reliabilityTest-retest reliability
เชน Pearson’s correlationInter-rater reliability
เชน Pearson’s correlation
![Page 10: Health Policy Evaluation - Ministry of Public Healthbps.moph.go.th/new_bps/sites/default/files/Health... · Program evaluation A systematic method for collecting, analyzing, and using](https://reader033.fdocuments.net/reader033/viewer/2022053004/5f086d377e708231d421f48c/html5/thumbnails/10.jpg)
10
คณสมบตของเครองมอประเมนทางสขภาพ(ตอ)
ความตรง (Validity) = ความสามารถของเครองมอในการแสดงผลในสงทตองการวดFace validityContent validityCriterion validity
Concurrent validity, predictive validity, postdictive validityConstruct validity
![Page 11: Health Policy Evaluation - Ministry of Public Healthbps.moph.go.th/new_bps/sites/default/files/Health... · Program evaluation A systematic method for collecting, analyzing, and using](https://reader033.fdocuments.net/reader033/viewer/2022053004/5f086d377e708231d421f48c/html5/thumbnails/11.jpg)
11
ดชนชวดขนาดปญหาสขภาพ
อบตการณ (Incidence) = จ านวน case ใหม ตอประชากรกลมเสยง
ความชก (Prevalence) = จ านวน case เกา +ใหม ตอประชากรทงหมด
![Page 12: Health Policy Evaluation - Ministry of Public Healthbps.moph.go.th/new_bps/sites/default/files/Health... · Program evaluation A systematic method for collecting, analyzing, and using](https://reader033.fdocuments.net/reader033/viewer/2022053004/5f086d377e708231d421f48c/html5/thumbnails/12.jpg)
Assessment of program theory
The program theory, also called a logic
model or impact pathway, is an
assumption, implicit in the way the
program is designed, about how the
program's actions are supposed to achieve the outcomes it intends.
12
Source: Centers for Disease Control and Prevention. Framework for Program Evaluation in Public Health. MMWR 1999;48(No. RR-11).
![Page 13: Health Policy Evaluation - Ministry of Public Healthbps.moph.go.th/new_bps/sites/default/files/Health... · Program evaluation A systematic method for collecting, analyzing, and using](https://reader033.fdocuments.net/reader033/viewer/2022053004/5f086d377e708231d421f48c/html5/thumbnails/13.jpg)
Assessment of program theory (cont.)
Assessment in relation to social needs
Assessment of logic and plausibility
Assessment through comparison with research and practice
Assessment via preliminary observation
13
![Page 14: Health Policy Evaluation - Ministry of Public Healthbps.moph.go.th/new_bps/sites/default/files/Health... · Program evaluation A systematic method for collecting, analyzing, and using](https://reader033.fdocuments.net/reader033/viewer/2022053004/5f086d377e708231d421f48c/html5/thumbnails/14.jpg)
กรอบแนวคด PRECEDE PROCEED Model (Green & Kreuter 1999)
![Page 15: Health Policy Evaluation - Ministry of Public Healthbps.moph.go.th/new_bps/sites/default/files/Health... · Program evaluation A systematic method for collecting, analyzing, and using](https://reader033.fdocuments.net/reader033/viewer/2022053004/5f086d377e708231d421f48c/html5/thumbnails/15.jpg)
การวเคราะหนโยบายทางเลอกทางการจดการปญหาการใชความรนแรงในนกเรยนอาชวศกษาโดยใชกรอบแนวคด PRECEDE Sub-model
![Page 16: Health Policy Evaluation - Ministry of Public Healthbps.moph.go.th/new_bps/sites/default/files/Health... · Program evaluation A systematic method for collecting, analyzing, and using](https://reader033.fdocuments.net/reader033/viewer/2022053004/5f086d377e708231d421f48c/html5/thumbnails/16.jpg)
การวเคราะหนโยบายทางเลอกทางการจดการปญหาการใชความรนแรงในนกเรยนอาชวศกษาโดยใชกรอบแนวคด PROCEED Sub-model
![Page 17: Health Policy Evaluation - Ministry of Public Healthbps.moph.go.th/new_bps/sites/default/files/Health... · Program evaluation A systematic method for collecting, analyzing, and using](https://reader033.fdocuments.net/reader033/viewer/2022053004/5f086d377e708231d421f48c/html5/thumbnails/17.jpg)
Socio-ecological model
![Page 18: Health Policy Evaluation - Ministry of Public Healthbps.moph.go.th/new_bps/sites/default/files/Health... · Program evaluation A systematic method for collecting, analyzing, and using](https://reader033.fdocuments.net/reader033/viewer/2022053004/5f086d377e708231d421f48c/html5/thumbnails/18.jpg)
Theories for Health Promotion Programs
Individual health behavior Interpersonal health behavior) Organization and community health behavior
18
![Page 19: Health Policy Evaluation - Ministry of Public Healthbps.moph.go.th/new_bps/sites/default/files/Health... · Program evaluation A systematic method for collecting, analyzing, and using](https://reader033.fdocuments.net/reader033/viewer/2022053004/5f086d377e708231d421f48c/html5/thumbnails/19.jpg)
ทฤษฎระดบบคคล
Health Belief Model Theory of Reasoned Action Transtheoretical Model and Stages of Change Precaution Adoption Process Model etc.
19
![Page 20: Health Policy Evaluation - Ministry of Public Healthbps.moph.go.th/new_bps/sites/default/files/Health... · Program evaluation A systematic method for collecting, analyzing, and using](https://reader033.fdocuments.net/reader033/viewer/2022053004/5f086d377e708231d421f48c/html5/thumbnails/20.jpg)
ทฤษฎระดบระหวางบคคล
Social Cognitive Theory Social Network Theory Transactional Model of Stress Interdependence Theory etc.
20
![Page 21: Health Policy Evaluation - Ministry of Public Healthbps.moph.go.th/new_bps/sites/default/files/Health... · Program evaluation A systematic method for collecting, analyzing, and using](https://reader033.fdocuments.net/reader033/viewer/2022053004/5f086d377e708231d421f48c/html5/thumbnails/21.jpg)
ทฤษฎระดบองคกรและชมชน
Community Building Community Organizing Diffusion of Innovations Communication Theories etc.
21
![Page 22: Health Policy Evaluation - Ministry of Public Healthbps.moph.go.th/new_bps/sites/default/files/Health... · Program evaluation A systematic method for collecting, analyzing, and using](https://reader033.fdocuments.net/reader033/viewer/2022053004/5f086d377e708231d421f48c/html5/thumbnails/22.jpg)
Health Belief Model) 22
ตระหนกถงความเสยงตอโรค
(Perceived Susceptibility)
- อาย เพศ เชอชาต- บคลกภาพ- ฐานะทางเศรษฐกจสงคม - ความร
ตระหนกถงภยคกคาม
(Perceived Threat)
สงกระตนใหเกดการกระท า
(Cues to Action)
การตระหนกถงผลดและผลเสยของการเปลยนพฤตกรรม
(Perceived Benefits versus Barrier to Behavioral Change)
แนวโนมของการเปลยนพฤตกรรม
(Likelihood of Behavioral Change)
การรบรสวนบคคล(Individual Perception)
ปจจยเสรม(Modifying Factors)
แนวโนมของการกระท า(Likelihood of Action)
Rosenstock in Egger et al 1990 :24
![Page 23: Health Policy Evaluation - Ministry of Public Healthbps.moph.go.th/new_bps/sites/default/files/Health... · Program evaluation A systematic method for collecting, analyzing, and using](https://reader033.fdocuments.net/reader033/viewer/2022053004/5f086d377e708231d421f48c/html5/thumbnails/23.jpg)
23The Transtheoretical Model of Behavioral Change
![Page 24: Health Policy Evaluation - Ministry of Public Healthbps.moph.go.th/new_bps/sites/default/files/Health... · Program evaluation A systematic method for collecting, analyzing, and using](https://reader033.fdocuments.net/reader033/viewer/2022053004/5f086d377e708231d421f48c/html5/thumbnails/24.jpg)
24ตวอยางการแกปญหาทางสขภาพจตของวยรนแบบใชทฤษฎ/หลกฐานทางวทยาศาสตร
สมาธสนในชนประถม
การเรยนดอย
ขาดการยอมรบจากกลมเพอน
กาวราว
ขอายใชสารเสพตดในวยรน
ซมเศราฆาตวตาย
คนหา proximal targets ในเดก
ประถมและใหการแกไขเพอปองกน distal
outcomes ในเดกวยรนSources: e.g., Kellam et al. (1991) Am J Community Psychol.
Aug;19(4):563-84; Kellam et al. (1989) NIDA Res Monogr.;95:368-9.
พฤตกรรมตอตาน
กฎระเบยบในวยรน
![Page 25: Health Policy Evaluation - Ministry of Public Healthbps.moph.go.th/new_bps/sites/default/files/Health... · Program evaluation A systematic method for collecting, analyzing, and using](https://reader033.fdocuments.net/reader033/viewer/2022053004/5f086d377e708231d421f48c/html5/thumbnails/25.jpg)
The Logical Framework Matrix (Logframe)25
Source: BARRETO (2010) cited in Leonellha Barreto Dillon Logical Framework Approach retrived from http://www.sswm.info/content/logical-framework-approach
![Page 26: Health Policy Evaluation - Ministry of Public Healthbps.moph.go.th/new_bps/sites/default/files/Health... · Program evaluation A systematic method for collecting, analyzing, and using](https://reader033.fdocuments.net/reader033/viewer/2022053004/5f086d377e708231d421f48c/html5/thumbnails/26.jpg)
Logframe example26
![Page 27: Health Policy Evaluation - Ministry of Public Healthbps.moph.go.th/new_bps/sites/default/files/Health... · Program evaluation A systematic method for collecting, analyzing, and using](https://reader033.fdocuments.net/reader033/viewer/2022053004/5f086d377e708231d421f48c/html5/thumbnails/27.jpg)
Assessing implementation
This evaluation determines whether the components identified as critical to the success of the program are being implemented.
27
Source: Rossi, P., Lipsey, M.W., & Freeman, H.E. (2004). Evaluation: a systematic approach (7th ed.). Thousand Oaks: Sage.
![Page 28: Health Policy Evaluation - Ministry of Public Healthbps.moph.go.th/new_bps/sites/default/files/Health... · Program evaluation A systematic method for collecting, analyzing, and using](https://reader033.fdocuments.net/reader033/viewer/2022053004/5f086d377e708231d421f48c/html5/thumbnails/28.jpg)
Questions of implementation evaluation
Finding out about service use
Finding out about service delivery
Finding out about the organization of the program
Looking at the program’s resources
Looking at participants’ experiences of the program §
28
Source: Purdon, S. et.al. (2001) RESEARCH METHODS FOR POLICY EVALUATION, the Department Work and Pensions,
![Page 29: Health Policy Evaluation - Ministry of Public Healthbps.moph.go.th/new_bps/sites/default/files/Health... · Program evaluation A systematic method for collecting, analyzing, and using](https://reader033.fdocuments.net/reader033/viewer/2022053004/5f086d377e708231d421f48c/html5/thumbnails/29.jpg)
Methods of implementation evaluation
Monitoring and Operational Research
Social research
Quantitative surveys
Qualitative research: depth interviews and group discussions
Case studies
29
Source: Purdon, S. et.al. (2001) RESEARCH METHODS FOR POLICY EVALUATION, the Department Work and Pensions,
![Page 30: Health Policy Evaluation - Ministry of Public Healthbps.moph.go.th/new_bps/sites/default/files/Health... · Program evaluation A systematic method for collecting, analyzing, and using](https://reader033.fdocuments.net/reader033/viewer/2022053004/5f086d377e708231d421f48c/html5/thumbnails/30.jpg)
Assessing the outcome/impact (effectiveness)
This involves trying to measure if the program has achieved its intended outcomesOutcomes are the factors that the program
is expected to affect
Impact Evaluation: to measure the impact a policy or program has on defined outcome measures. It usually involves measuring the counterfactual
30
Source: Rossi, P., Lipsey, M.W., & Freeman, H.E. (2004). Evaluation: a systematic approach (7th ed.). Thousand Oaks: Sage.
Purdon, S. et.al. (2001) RESEARCH METHODS FOR POLICY EVALUATION, the Department Work and Pensions,
![Page 31: Health Policy Evaluation - Ministry of Public Healthbps.moph.go.th/new_bps/sites/default/files/Health... · Program evaluation A systematic method for collecting, analyzing, and using](https://reader033.fdocuments.net/reader033/viewer/2022053004/5f086d377e708231d421f48c/html5/thumbnails/31.jpg)
Program evaluation: level of outcomes 31
Efficacy
Effectiveness
Sustainability
![Page 32: Health Policy Evaluation - Ministry of Public Healthbps.moph.go.th/new_bps/sites/default/files/Health... · Program evaluation A systematic method for collecting, analyzing, and using](https://reader033.fdocuments.net/reader033/viewer/2022053004/5f086d377e708231d421f48c/html5/thumbnails/32.jpg)
Methods of outcome/impact evaluation
Randomized trials – an experimental design
Matched area comparison design
Before-after study
Matched comparison group design
32
Source: Purdon, S. et.al. (2001) RESEARCH METHODS FOR POLICY EVALUATION, the Department Work and Pensions,
![Page 33: Health Policy Evaluation - Ministry of Public Healthbps.moph.go.th/new_bps/sites/default/files/Health... · Program evaluation A systematic method for collecting, analyzing, and using](https://reader033.fdocuments.net/reader033/viewer/2022053004/5f086d377e708231d421f48c/html5/thumbnails/33.jpg)
Assessing efficiency
cost-benefit or cost-efficiency analysis assesses the efficiency of a program
33
Source: Rossi, P., Lipsey, M.W., & Freeman, H.E. (2004). Evaluation: a systematic approach (7th ed.). Thousand Oaks: Sage.
![Page 34: Health Policy Evaluation - Ministry of Public Healthbps.moph.go.th/new_bps/sites/default/files/Health... · Program evaluation A systematic method for collecting, analyzing, and using](https://reader033.fdocuments.net/reader033/viewer/2022053004/5f086d377e708231d421f48c/html5/thumbnails/34.jpg)
CIPP Model:
Context: What needs to be done? Vs. Were important needs addressed?
Input: How should it be done? Vs. Was a defensible design employed?
Process: Is it being done? Vs. Was the design well executed?
Product: Is it succeeding? Vs. Did the effort succeed?
34
![Page 35: Health Policy Evaluation - Ministry of Public Healthbps.moph.go.th/new_bps/sites/default/files/Health... · Program evaluation A systematic method for collecting, analyzing, and using](https://reader033.fdocuments.net/reader033/viewer/2022053004/5f086d377e708231d421f48c/html5/thumbnails/35.jpg)
CDC’s Framework for program evaluation in public health
35
Centers for Disease Control and Prevention. Framework for program evaluation in public health. MMWR 1999;48 (No. RR-11)
![Page 36: Health Policy Evaluation - Ministry of Public Healthbps.moph.go.th/new_bps/sites/default/files/Health... · Program evaluation A systematic method for collecting, analyzing, and using](https://reader033.fdocuments.net/reader033/viewer/2022053004/5f086d377e708231d421f48c/html5/thumbnails/36.jpg)
Five-tiered approach
Tier 1: needs assessment (sometimes referred to as pre-implementation)[29]
Tier 2: monitoring and accountability
Tier 3: quality review and program clarification (sometimes referred to as understanding and refining)[30]
Tier 4: achieving outcomes
Tier 5: establishing impact
36
![Page 37: Health Policy Evaluation - Ministry of Public Healthbps.moph.go.th/new_bps/sites/default/files/Health... · Program evaluation A systematic method for collecting, analyzing, and using](https://reader033.fdocuments.net/reader033/viewer/2022053004/5f086d377e708231d421f48c/html5/thumbnails/37.jpg)
Case studies
Background and significance
Evaluation questions and objectives
Evaluation Framework
Methods
Results and Discussions
37