WHO VIP Webinar 2011 1 Evaluation Considerations: Measures & Methods Shrikant I. Bangdiwala, PhD...
-
Upload
austin-blevins -
Category
Documents
-
view
223 -
download
1
Transcript of WHO VIP Webinar 2011 1 Evaluation Considerations: Measures & Methods Shrikant I. Bangdiwala, PhD...
![Page 1: WHO VIP Webinar 2011 1 Evaluation Considerations: Measures & Methods Shrikant I. Bangdiwala, PhD Professor of Research in Biostatistics Injury Prevention.](https://reader035.fdocuments.net/reader035/viewer/2022062619/55153a625503465e608b5dbd/html5/thumbnails/1.jpg)
WHO VIP Webinar 20111
Evaluation Considerations: Measures & Methods
Shrikant I. Bangdiwala, PhD
Professor of Research in BiostatisticsInjury Prevention Research CenterUniversity of North Carolina at Chapel Hill, USA
presented at
![Page 2: WHO VIP Webinar 2011 1 Evaluation Considerations: Measures & Methods Shrikant I. Bangdiwala, PhD Professor of Research in Biostatistics Injury Prevention.](https://reader035.fdocuments.net/reader035/viewer/2022062619/55153a625503465e608b5dbd/html5/thumbnails/2.jpg)
WHO VIP Webinar 20112
Content
Purpose of evaluation Cycle of program planning & evaluation Indicators Study designs Statistical modeling Challenges
![Page 3: WHO VIP Webinar 2011 1 Evaluation Considerations: Measures & Methods Shrikant I. Bangdiwala, PhD Professor of Research in Biostatistics Injury Prevention.](https://reader035.fdocuments.net/reader035/viewer/2022062619/55153a625503465e608b5dbd/html5/thumbnails/3.jpg)
Safety 2010 London3
What are we ‘evaluating’?
Actions, programs, activities Conducted in a community setting, over a
period of time Aimed at reducing deaths, injuries, and/or
events and behaviors that cause injuries
WHO VIP Webinar 20113
![Page 4: WHO VIP Webinar 2011 1 Evaluation Considerations: Measures & Methods Shrikant I. Bangdiwala, PhD Professor of Research in Biostatistics Injury Prevention.](https://reader035.fdocuments.net/reader035/viewer/2022062619/55153a625503465e608b5dbd/html5/thumbnails/4.jpg)
WHO VIP Webinar 20114
Example: Suwon, South Korea
area of ‘safety promotion’
http://www.phs.ki.se/csp/safecom/suwon2.htm
![Page 5: WHO VIP Webinar 2011 1 Evaluation Considerations: Measures & Methods Shrikant I. Bangdiwala, PhD Professor of Research in Biostatistics Injury Prevention.](https://reader035.fdocuments.net/reader035/viewer/2022062619/55153a625503465e608b5dbd/html5/thumbnails/5.jpg)
WHO VIP Webinar 20115
Why do we ‘evaluate’?
To know ourselves what works and if we are doing some goodIn performing some activityIn the communityIn the country
To convince funders and supporters that their investment is worthwhile
To convince the community about the benefits of the multiple activities and actions carried out
![Page 6: WHO VIP Webinar 2011 1 Evaluation Considerations: Measures & Methods Shrikant I. Bangdiwala, PhD Professor of Research in Biostatistics Injury Prevention.](https://reader035.fdocuments.net/reader035/viewer/2022062619/55153a625503465e608b5dbd/html5/thumbnails/6.jpg)
WHO VIP Webinar 20116 NSC Chicago 2010 6
Main purposes of evaluation
Evaluation helps determine:How well a program/policy works relative to its
goals & objectivesWhy a program/policy did or didn’t work, relative
to planned processHow to restructure a program/policy to make it
work, or work betterWhether to change funding for a program
![Page 7: WHO VIP Webinar 2011 1 Evaluation Considerations: Measures & Methods Shrikant I. Bangdiwala, PhD Professor of Research in Biostatistics Injury Prevention.](https://reader035.fdocuments.net/reader035/viewer/2022062619/55153a625503465e608b5dbd/html5/thumbnails/7.jpg)
WHO VIP Webinar 20117
Methodological complications
Multiplicities Multiple components of a programMultiple populations at riskMultiple study designsMultiple types of effects/impacts/outcomes &
severitiesMultiple audiences/objectives of ‘evaluation’Multiple methods for conducting evaluation
![Page 8: WHO VIP Webinar 2011 1 Evaluation Considerations: Measures & Methods Shrikant I. Bangdiwala, PhD Professor of Research in Biostatistics Injury Prevention.](https://reader035.fdocuments.net/reader035/viewer/2022062619/55153a625503465e608b5dbd/html5/thumbnails/8.jpg)
WHO VIP Webinar 20118 8
When should evaluation be considered? Evaluation needs to begin in, and be part of,
the planning process… Otherwise, “if you do not know where you are
going, it does not matter which way you go, and you will never know if you got there or not!”
Lewis Carroll (1872)
Alice in Wonderland
Adapted from M. Garrettson
![Page 9: WHO VIP Webinar 2011 1 Evaluation Considerations: Measures & Methods Shrikant I. Bangdiwala, PhD Professor of Research in Biostatistics Injury Prevention.](https://reader035.fdocuments.net/reader035/viewer/2022062619/55153a625503465e608b5dbd/html5/thumbnails/9.jpg)
WHO VIP Webinar 20119 NSC Chicago 2010 9
ProgramPlanningPhase
Formative EvaluationHow can the program activities be improved before
implementation?
Process EvaluationHow is/was the program (being) implemented?
PostProgramPhase
Impact / OutcomeDid the program succeed in achieving the intended
impact or outcome?
Program Implementation
Phase
Types of evaluation depending on program phase
![Page 10: WHO VIP Webinar 2011 1 Evaluation Considerations: Measures & Methods Shrikant I. Bangdiwala, PhD Professor of Research in Biostatistics Injury Prevention.](https://reader035.fdocuments.net/reader035/viewer/2022062619/55153a625503465e608b5dbd/html5/thumbnails/10.jpg)
WHO VIP Webinar 201110
Cycle of program planning and evaluation
Adapted from C Runyan
![Page 11: WHO VIP Webinar 2011 1 Evaluation Considerations: Measures & Methods Shrikant I. Bangdiwala, PhD Professor of Research in Biostatistics Injury Prevention.](https://reader035.fdocuments.net/reader035/viewer/2022062619/55153a625503465e608b5dbd/html5/thumbnails/11.jpg)
WHO VIP Webinar 201111 11
Identify population & problem Surveillance data Other needs assessment strategies
key informant interviewsfocus groupssurveys evaluations of past programs literatureconsultation with peersother info…
![Page 12: WHO VIP Webinar 2011 1 Evaluation Considerations: Measures & Methods Shrikant I. Bangdiwala, PhD Professor of Research in Biostatistics Injury Prevention.](https://reader035.fdocuments.net/reader035/viewer/2022062619/55153a625503465e608b5dbd/html5/thumbnails/12.jpg)
WHO VIP Webinar 201112 NSC Chicago 2010 12
Test & refineimplementation
Test, Refine,Implement
Choose strategies
Set goals/objectives
Disseminate
Identifyproblem
& population
Define target audience
Evaluation: •Formative
Implement
Identify resources
Evaluation:•Process•Impact•Outcome
![Page 13: WHO VIP Webinar 2011 1 Evaluation Considerations: Measures & Methods Shrikant I. Bangdiwala, PhD Professor of Research in Biostatistics Injury Prevention.](https://reader035.fdocuments.net/reader035/viewer/2022062619/55153a625503465e608b5dbd/html5/thumbnails/13.jpg)
WHO VIP Webinar 201113 13
Define target audience
To whom is the program directed?Whose injuries need to be reduced?Who is the target of the program?
• at risk persons• care givers (e.g. parents)• general public• media• decision makers
![Page 14: WHO VIP Webinar 2011 1 Evaluation Considerations: Measures & Methods Shrikant I. Bangdiwala, PhD Professor of Research in Biostatistics Injury Prevention.](https://reader035.fdocuments.net/reader035/viewer/2022062619/55153a625503465e608b5dbd/html5/thumbnails/14.jpg)
WHO VIP Webinar 201114 14
Understand target audience
What are their characteristics?Special needs (e.g. literacy)Interests, concerns, prioritiesAttitudes & beliefs re: problem & solutions to
problem Cultural issues
![Page 15: WHO VIP Webinar 2011 1 Evaluation Considerations: Measures & Methods Shrikant I. Bangdiwala, PhD Professor of Research in Biostatistics Injury Prevention.](https://reader035.fdocuments.net/reader035/viewer/2022062619/55153a625503465e608b5dbd/html5/thumbnails/15.jpg)
WHO VIP Webinar 201115 15
Test & refineimplementation
Test, Refine,Implement
Choose strategies
Set goals/objectives
Disseminate
Identifyproblem
& population
Define target audience
Evaluation: •Formative
Implement
Identify resources
Evaluation:•Process•Impact•Outcome
![Page 16: WHO VIP Webinar 2011 1 Evaluation Considerations: Measures & Methods Shrikant I. Bangdiwala, PhD Professor of Research in Biostatistics Injury Prevention.](https://reader035.fdocuments.net/reader035/viewer/2022062619/55153a625503465e608b5dbd/html5/thumbnails/16.jpg)
WHO VIP Webinar 201116 16
Identify resources
Community partnersinterest in topicworking on similar projects
On-going activities Sources of financial support Interests in community
![Page 17: WHO VIP Webinar 2011 1 Evaluation Considerations: Measures & Methods Shrikant I. Bangdiwala, PhD Professor of Research in Biostatistics Injury Prevention.](https://reader035.fdocuments.net/reader035/viewer/2022062619/55153a625503465e608b5dbd/html5/thumbnails/17.jpg)
WHO VIP Webinar 201117 17
Test & refineimplementation
Test, Refine,Implement
Choose strategies
Set goals/objectives
Disseminate
Identifyproblem
& population
Define target audience
Evaluation: •Formative
Implement
Identify resources
Evaluation:•Process•Impact•Outcome
![Page 18: WHO VIP Webinar 2011 1 Evaluation Considerations: Measures & Methods Shrikant I. Bangdiwala, PhD Professor of Research in Biostatistics Injury Prevention.](https://reader035.fdocuments.net/reader035/viewer/2022062619/55153a625503465e608b5dbd/html5/thumbnails/18.jpg)
WHO VIP Webinar 201118 18
Set goals & objectives
Goalbroad statement of what program is trying to
accomplish
ObjectivesSpecificMeasurableTime-framed
![Page 19: WHO VIP Webinar 2011 1 Evaluation Considerations: Measures & Methods Shrikant I. Bangdiwala, PhD Professor of Research in Biostatistics Injury Prevention.](https://reader035.fdocuments.net/reader035/viewer/2022062619/55153a625503465e608b5dbd/html5/thumbnails/19.jpg)
WHO VIP Webinar 201119 19
Test & refineimplementation
Test, Refine,Implement
Choose strategies
Set goals/objectives
Disseminate
Identifyproblem
& population
Define target audience
Evaluation: •Formative
Implement
Identify resources
Evaluation:•Process•Impact•Outcome
![Page 20: WHO VIP Webinar 2011 1 Evaluation Considerations: Measures & Methods Shrikant I. Bangdiwala, PhD Professor of Research in Biostatistics Injury Prevention.](https://reader035.fdocuments.net/reader035/viewer/2022062619/55153a625503465e608b5dbd/html5/thumbnails/20.jpg)
WHO VIP Webinar 201120 20
Choose Strategies
Identify existing strategies/programsLiterature: evidence based? promising practice?WHO manualsSuccesses from other communities-regions-
countries Develop new strategies:
Logic model (how would it work)Haddon matrix
![Page 21: WHO VIP Webinar 2011 1 Evaluation Considerations: Measures & Methods Shrikant I. Bangdiwala, PhD Professor of Research in Biostatistics Injury Prevention.](https://reader035.fdocuments.net/reader035/viewer/2022062619/55153a625503465e608b5dbd/html5/thumbnails/21.jpg)
WHO VIP Webinar 201121 21
Haddon Matrix
Person Vehicle/vector
Physical Environ.
Social Environ.
Pre-event
Event
Post-event
Haddon 1970 Am J Public Health
![Page 22: WHO VIP Webinar 2011 1 Evaluation Considerations: Measures & Methods Shrikant I. Bangdiwala, PhD Professor of Research in Biostatistics Injury Prevention.](https://reader035.fdocuments.net/reader035/viewer/2022062619/55153a625503465e608b5dbd/html5/thumbnails/22.jpg)
WHO VIP Webinar 201122 22
Pre-Event Other??
Decision CriteriaDecision Criteria
PhasesPhases
3-dimensional Haddon Matrix
P
erso
n
Veh
icle
/V
ecto
r)
P
hysi
cal
Env
iron.
Soc
ial
Env
iron.
FactorsFactors
Event
Post-event
FeasibilityPreferences
StigmatizationEquity
FreedomCost
Effectiveness
Runyan 1998 Injury Prevention
![Page 23: WHO VIP Webinar 2011 1 Evaluation Considerations: Measures & Methods Shrikant I. Bangdiwala, PhD Professor of Research in Biostatistics Injury Prevention.](https://reader035.fdocuments.net/reader035/viewer/2022062619/55153a625503465e608b5dbd/html5/thumbnails/23.jpg)
WHO VIP Webinar 201123 NSC Chicago 2010 23
Test & refineimplementation
Test, Refine,Implement
Choose strategies
Set goals/objectives
Disseminate
Identifyproblem
& population
Define target audience
Evaluation: •Formative
Implement
Identify resources
Evaluation:•Process•Impact•Outcome
![Page 24: WHO VIP Webinar 2011 1 Evaluation Considerations: Measures & Methods Shrikant I. Bangdiwala, PhD Professor of Research in Biostatistics Injury Prevention.](https://reader035.fdocuments.net/reader035/viewer/2022062619/55153a625503465e608b5dbd/html5/thumbnails/24.jpg)
WHO VIP Webinar 201124 24
Formative Evaluation
What is the best way to influence the target population?
Will the activities reach the people intended, be understood and accepted by target population?
How can activities be improved?
Improves (pilot-tests) program activities before full-scale implementation
May increase likelihood program or policy will succeed
May help stretch resources
Why it’s usefulQuestions it answers
* Modified from Thompson & McClintock, 2000
![Page 25: WHO VIP Webinar 2011 1 Evaluation Considerations: Measures & Methods Shrikant I. Bangdiwala, PhD Professor of Research in Biostatistics Injury Prevention.](https://reader035.fdocuments.net/reader035/viewer/2022062619/55153a625503465e608b5dbd/html5/thumbnails/25.jpg)
WHO VIP Webinar 201125 NSC Chicago 2010 25
Test & refineimplementation
Test, Refine,Implement
Choose strategies
Set goals/objectives
Disseminate
Identifyproblem
& population
Define target audience
Evaluation: •Formative
Implement
Identify resources
Evaluation:•Process•Impact•Outcome
![Page 26: WHO VIP Webinar 2011 1 Evaluation Considerations: Measures & Methods Shrikant I. Bangdiwala, PhD Professor of Research in Biostatistics Injury Prevention.](https://reader035.fdocuments.net/reader035/viewer/2022062619/55153a625503465e608b5dbd/html5/thumbnails/26.jpg)
WHO VIP Webinar 201126 26
Implementation
As planned, with attention to detail
Documented clearly so others can replicate if appropriate
![Page 27: WHO VIP Webinar 2011 1 Evaluation Considerations: Measures & Methods Shrikant I. Bangdiwala, PhD Professor of Research in Biostatistics Injury Prevention.](https://reader035.fdocuments.net/reader035/viewer/2022062619/55153a625503465e608b5dbd/html5/thumbnails/27.jpg)
WHO VIP Webinar 201127 NSC Chicago 2010 27
Test & refineimplementation
Test, Refine,Implement
Choose strategies
Set goals/objectives
Disseminate
Identifyproblem
& population
Define target audience
Evaluation: •Formative
Implement
Identify resources
Evaluation:•Process•Impact•Outcome
![Page 28: WHO VIP Webinar 2011 1 Evaluation Considerations: Measures & Methods Shrikant I. Bangdiwala, PhD Professor of Research in Biostatistics Injury Prevention.](https://reader035.fdocuments.net/reader035/viewer/2022062619/55153a625503465e608b5dbd/html5/thumbnails/28.jpg)
WHO VIP Webinar 201128 28
Process evaluation
Purpose is to address:What was done?How was it implemented?How well was it implemented?Was it implemented as planned?
![Page 29: WHO VIP Webinar 2011 1 Evaluation Considerations: Measures & Methods Shrikant I. Bangdiwala, PhD Professor of Research in Biostatistics Injury Prevention.](https://reader035.fdocuments.net/reader035/viewer/2022062619/55153a625503465e608b5dbd/html5/thumbnails/29.jpg)
WHO VIP Webinar 201129 29
Process evaluation – examples of questions
• Who carried out intervention?• Was this the appropriate person/group?• Who supported and opposed intervention?• What methods/activities were used?
![Page 30: WHO VIP Webinar 2011 1 Evaluation Considerations: Measures & Methods Shrikant I. Bangdiwala, PhD Professor of Research in Biostatistics Injury Prevention.](https://reader035.fdocuments.net/reader035/viewer/2022062619/55153a625503465e608b5dbd/html5/thumbnails/30.jpg)
WHO VIP Webinar 201130 30
Process evaluation - why is it useful?
• Allows replication of programs that work.• Helps understand why programs fail.
* Modified from Thompson & McClintock, 2000
![Page 31: WHO VIP Webinar 2011 1 Evaluation Considerations: Measures & Methods Shrikant I. Bangdiwala, PhD Professor of Research in Biostatistics Injury Prevention.](https://reader035.fdocuments.net/reader035/viewer/2022062619/55153a625503465e608b5dbd/html5/thumbnails/31.jpg)
WHO VIP Webinar 201131 31
The intervention cannot be a black box…
OutcomeIdea ?
It must be clearly understood
![Page 32: WHO VIP Webinar 2011 1 Evaluation Considerations: Measures & Methods Shrikant I. Bangdiwala, PhD Professor of Research in Biostatistics Injury Prevention.](https://reader035.fdocuments.net/reader035/viewer/2022062619/55153a625503465e608b5dbd/html5/thumbnails/32.jpg)
WHO VIP Webinar 201132 NSC Chicago 2010 32
Test & refineimplementation
Test, Refine,Implement
Choose strategies
Set goals/objectives
Disseminate
Identifyproblem
& population
Define target audience
Evaluation: •Formative
Implement
Identify resources
Evaluation:•Process•Impact•Outcome
![Page 33: WHO VIP Webinar 2011 1 Evaluation Considerations: Measures & Methods Shrikant I. Bangdiwala, PhD Professor of Research in Biostatistics Injury Prevention.](https://reader035.fdocuments.net/reader035/viewer/2022062619/55153a625503465e608b5dbd/html5/thumbnails/33.jpg)
WHO VIP Webinar 201133 33
Impact evaluation
Purpose is to address changes in: knowledge attitudes beliefs/ values skills behaviors / practices
![Page 34: WHO VIP Webinar 2011 1 Evaluation Considerations: Measures & Methods Shrikant I. Bangdiwala, PhD Professor of Research in Biostatistics Injury Prevention.](https://reader035.fdocuments.net/reader035/viewer/2022062619/55153a625503465e608b5dbd/html5/thumbnails/34.jpg)
WHO VIP Webinar 201134 34
Using impact measures for
Establishing effectivenessSuppose we have a public safety campaign as our
strategyNeed to show Campaign Behavior Outcome
If we already have demonstrated that Behavior Outcome
We simply need to show Campaign Behavior
![Page 35: WHO VIP Webinar 2011 1 Evaluation Considerations: Measures & Methods Shrikant I. Bangdiwala, PhD Professor of Research in Biostatistics Injury Prevention.](https://reader035.fdocuments.net/reader035/viewer/2022062619/55153a625503465e608b5dbd/html5/thumbnails/35.jpg)
WHO VIP Webinar 201135 35
Outcome evaluation
Purpose is to address changes in:injury events (e.g. frequency, type, pattern) morbidity (e.g. frequency, severity, type)mortality (e.g. frequency, time to death)cost (e.g. direct and indirect)
![Page 36: WHO VIP Webinar 2011 1 Evaluation Considerations: Measures & Methods Shrikant I. Bangdiwala, PhD Professor of Research in Biostatistics Injury Prevention.](https://reader035.fdocuments.net/reader035/viewer/2022062619/55153a625503465e608b5dbd/html5/thumbnails/36.jpg)
WHO VIP Webinar 201136 NSC Chicago 2010 36
OutcomesImpacts Intervention
Physician counseling
parents
Enforcement of helmet law
Media campaign
Parental attitudes
toward child helmet use
Purchase of helmets
Use of helmets by children
Head injury in bike crashes
Deaths from head injury in crashes
Example: Bike helmets
![Page 37: WHO VIP Webinar 2011 1 Evaluation Considerations: Measures & Methods Shrikant I. Bangdiwala, PhD Professor of Research in Biostatistics Injury Prevention.](https://reader035.fdocuments.net/reader035/viewer/2022062619/55153a625503465e608b5dbd/html5/thumbnails/37.jpg)
WHO VIP Webinar 201137 37
Evaluation examples of questions for local policy of smoke alarms
Did the local policy of smoke alarms in apartments… Get passed Where people aware of it? Did people have access to smoke alarms? Did people get them installed properly? Do people keep them maintained? Lead to a reduction in the number or rates of:
events (e.g. apartment fires) injuries deaths costs (e.g. burn center costs, family burden, property loss)
![Page 38: WHO VIP Webinar 2011 1 Evaluation Considerations: Measures & Methods Shrikant I. Bangdiwala, PhD Professor of Research in Biostatistics Injury Prevention.](https://reader035.fdocuments.net/reader035/viewer/2022062619/55153a625503465e608b5dbd/html5/thumbnails/38.jpg)
WHO VIP Webinar 2011383838
Evaluation – selection of measures‘Quantitative Indicators’
Process Impact Outcome
Health related Financial
![Page 39: WHO VIP Webinar 2011 1 Evaluation Considerations: Measures & Methods Shrikant I. Bangdiwala, PhD Professor of Research in Biostatistics Injury Prevention.](https://reader035.fdocuments.net/reader035/viewer/2022062619/55153a625503465e608b5dbd/html5/thumbnails/39.jpg)
WHO VIP Webinar 201139
Choice of measure or indicator
We need to choose appropriate impact and outcome measures ‘Soft’ (more difficult to measure) outcomes –
Perceptions constructs: fear, insecurity, wellbeing, quality of life Knowledge, attitudes and behaviors constructs
Hard outcomes – Deaths, hospitalizations, disabilities due to injuries and violence Societal impacts – local development indicators
Economics outcomes – Direct $/€/£/¥, indirect DALYs, QALYs, opportunities lost, burdens
![Page 40: WHO VIP Webinar 2011 1 Evaluation Considerations: Measures & Methods Shrikant I. Bangdiwala, PhD Professor of Research in Biostatistics Injury Prevention.](https://reader035.fdocuments.net/reader035/viewer/2022062619/55153a625503465e608b5dbd/html5/thumbnails/40.jpg)
WHO VIP Webinar 201140
Evidence of effectiveness
Obtain qualitative ‘evidence’ to complement the quantitative ‘evidence’Ex. Are “multisectorial collaborations and
partnerships” friendly and functioning well?Ex. Is “community participation” optimal?
Incorporate process indicators Incorporate narratives & testimonials
![Page 41: WHO VIP Webinar 2011 1 Evaluation Considerations: Measures & Methods Shrikant I. Bangdiwala, PhD Professor of Research in Biostatistics Injury Prevention.](https://reader035.fdocuments.net/reader035/viewer/2022062619/55153a625503465e608b5dbd/html5/thumbnails/41.jpg)
WHO VIP Webinar 201141 NSC Chicago 2010 41
Test & refineimplementation
Test, Refine,Implement
Choose strategies
Set goals/objectives
Disseminate
Identifyproblem
& population
Define target audience
Evaluation: •Formative
Implement
Identify resources
Evaluation:•Process•Impact•Outcome
![Page 42: WHO VIP Webinar 2011 1 Evaluation Considerations: Measures & Methods Shrikant I. Bangdiwala, PhD Professor of Research in Biostatistics Injury Prevention.](https://reader035.fdocuments.net/reader035/viewer/2022062619/55153a625503465e608b5dbd/html5/thumbnails/42.jpg)
WHO VIP Webinar 201142 42
Dissemination
Dissemination not done wellNot attemptedNot based on research about how to disseminate
information to intended audience Dissemination done well
Defining audienceHow to access audienceHow best to communicate change message to themPresentation of clear, straightforward messages
![Page 43: WHO VIP Webinar 2011 1 Evaluation Considerations: Measures & Methods Shrikant I. Bangdiwala, PhD Professor of Research in Biostatistics Injury Prevention.](https://reader035.fdocuments.net/reader035/viewer/2022062619/55153a625503465e608b5dbd/html5/thumbnails/43.jpg)
WHO VIP Webinar 2011434343
Evaluation measures
Lead to evidence of effectiveness But only if the research and study
methodologies, and the statistical analyses methodologies, are appropriate to convince the funders and supporters, the skeptics, the stakeholders, the communityand understandable
![Page 44: WHO VIP Webinar 2011 1 Evaluation Considerations: Measures & Methods Shrikant I. Bangdiwala, PhD Professor of Research in Biostatistics Injury Prevention.](https://reader035.fdocuments.net/reader035/viewer/2022062619/55153a625503465e608b5dbd/html5/thumbnails/44.jpg)
WHO VIP Webinar 201144
Research methodology approach:Evidence of effectiveness
Obtain quantitative ‘evidence’ that favors the hypothesis that the intervention is effective as opposed to the (null) hypothesis that the intervention is not effective.
How? Experimental study designs - randomized clinical trials,
grouped randomized experiments, community-randomized studies
Quasi-experimental study designs - non-randomized comparative studies, before-after studies
Observational studies - cohort studies, case-control studies and comparative cross-sectional studies
![Page 45: WHO VIP Webinar 2011 1 Evaluation Considerations: Measures & Methods Shrikant I. Bangdiwala, PhD Professor of Research in Biostatistics Injury Prevention.](https://reader035.fdocuments.net/reader035/viewer/2022062619/55153a625503465e608b5dbd/html5/thumbnails/45.jpg)
WHO VIP Webinar 2011454545
Randomized controlled trial (RCT) / Experiment
‘strongest’ evidence
Randomize
InterventionGroup
Control Group
O X O
O OX’
![Page 46: WHO VIP Webinar 2011 1 Evaluation Considerations: Measures & Methods Shrikant I. Bangdiwala, PhD Professor of Research in Biostatistics Injury Prevention.](https://reader035.fdocuments.net/reader035/viewer/2022062619/55153a625503465e608b5dbd/html5/thumbnails/46.jpg)
WHO VIP Webinar 2011464646
Quasi-experimental designs
‘qualified’ evidence
Intervention Group
Comparison Group
O X O
O O
![Page 47: WHO VIP Webinar 2011 1 Evaluation Considerations: Measures & Methods Shrikant I. Bangdiwala, PhD Professor of Research in Biostatistics Injury Prevention.](https://reader035.fdocuments.net/reader035/viewer/2022062619/55153a625503465e608b5dbd/html5/thumbnails/47.jpg)
WHO VIP Webinar 2011474747
One group pre/post
‘weak’ evidence
InterventionGroup
O X O
![Page 48: WHO VIP Webinar 2011 1 Evaluation Considerations: Measures & Methods Shrikant I. Bangdiwala, PhD Professor of Research in Biostatistics Injury Prevention.](https://reader035.fdocuments.net/reader035/viewer/2022062619/55153a625503465e608b5dbd/html5/thumbnails/48.jpg)
WHO VIP Webinar 2011484848
One group – multiple pre / multiple post
better ‘weak’ evidenceIntervention
GroupO O O X O O O O O
![Page 49: WHO VIP Webinar 2011 1 Evaluation Considerations: Measures & Methods Shrikant I. Bangdiwala, PhD Professor of Research in Biostatistics Injury Prevention.](https://reader035.fdocuments.net/reader035/viewer/2022062619/55153a625503465e608b5dbd/html5/thumbnails/49.jpg)
WHO VIP Webinar 2011494949
One group, post only
‘basically ignorable’ evidence
Intervention Group
X O
![Page 50: WHO VIP Webinar 2011 1 Evaluation Considerations: Measures & Methods Shrikant I. Bangdiwala, PhD Professor of Research in Biostatistics Injury Prevention.](https://reader035.fdocuments.net/reader035/viewer/2022062619/55153a625503465e608b5dbd/html5/thumbnails/50.jpg)
Safety 2010 London505050
Observational designs- cohort study
evidence?Self-chosen Intervention Group
Self-chosen Non-intervention Group
X X X
O O O
WHO VIP Webinar 201150
![Page 51: WHO VIP Webinar 2011 1 Evaluation Considerations: Measures & Methods Shrikant I. Bangdiwala, PhD Professor of Research in Biostatistics Injury Prevention.](https://reader035.fdocuments.net/reader035/viewer/2022062619/55153a625503465e608b5dbd/html5/thumbnails/51.jpg)
Safety 2010 London515151
Observational designs- case-control study
evidence? Cases
Controls
X
O
X
O
WHO VIP Webinar 201151
![Page 52: WHO VIP Webinar 2011 1 Evaluation Considerations: Measures & Methods Shrikant I. Bangdiwala, PhD Professor of Research in Biostatistics Injury Prevention.](https://reader035.fdocuments.net/reader035/viewer/2022062619/55153a625503465e608b5dbd/html5/thumbnails/52.jpg)
Safety 2010 London525252
Observational designs- cross-sectional study
evidence?
InjuredX X O XO
O X OO O
Non-injured
WHO VIP Webinar 201152
![Page 53: WHO VIP Webinar 2011 1 Evaluation Considerations: Measures & Methods Shrikant I. Bangdiwala, PhD Professor of Research in Biostatistics Injury Prevention.](https://reader035.fdocuments.net/reader035/viewer/2022062619/55153a625503465e608b5dbd/html5/thumbnails/53.jpg)
WHO VIP Webinar 201153
Statistical analysis methodologies
Choice - often guided by what has been done previously, or what is feasible to do, or easy to explain
Choice should be tailored to the audience & their ability to understand results; but also on the ability of the presenter to explain the methodologies
![Page 54: WHO VIP Webinar 2011 1 Evaluation Considerations: Measures & Methods Shrikant I. Bangdiwala, PhD Professor of Research in Biostatistics Injury Prevention.](https://reader035.fdocuments.net/reader035/viewer/2022062619/55153a625503465e608b5dbd/html5/thumbnails/54.jpg)
WHO VIP Webinar 201154
Statistical analysis Determined by research question(s) Guided by study design – experimental or observational
Group randomized controlled experiment Non-randomized comparison study Single site pre/post; surveillance study Retrospective or cross-sectional
Guided by whether outcome is studied at a single time point or multiple time points Time series analyses
Guided by audience Visual and descriptive appreciation
![Page 55: WHO VIP Webinar 2011 1 Evaluation Considerations: Measures & Methods Shrikant I. Bangdiwala, PhD Professor of Research in Biostatistics Injury Prevention.](https://reader035.fdocuments.net/reader035/viewer/2022062619/55153a625503465e608b5dbd/html5/thumbnails/55.jpg)
WHO VIP Webinar 201155
Visual and descriptive analysis – longitudinal time series
Example:
Espitia et al (2008) Salud Pública Mexico
![Page 56: WHO VIP Webinar 2011 1 Evaluation Considerations: Measures & Methods Shrikant I. Bangdiwala, PhD Professor of Research in Biostatistics Injury Prevention.](https://reader035.fdocuments.net/reader035/viewer/2022062619/55153a625503465e608b5dbd/html5/thumbnails/56.jpg)
Safety 2010 London56
Visual and descriptive analysis – comparisons over time
Example:
www.gapminder.org
WHO VIP Webinar 201156
![Page 57: WHO VIP Webinar 2011 1 Evaluation Considerations: Measures & Methods Shrikant I. Bangdiwala, PhD Professor of Research in Biostatistics Injury Prevention.](https://reader035.fdocuments.net/reader035/viewer/2022062619/55153a625503465e608b5dbd/html5/thumbnails/57.jpg)
WHO VIP Webinar 201157
Statistical analysis - challenge
But what we as a field have not done as well as other fields, is to draw strength from numbers develop collective evidence
Combine results from multiple studiesSystematic reviews (of observational studies)Meta analysis (of experimental & observational studies)Meta regression (of heterogeneous studies)Mixed treatment meta regression (for indirect
comparisons)
![Page 58: WHO VIP Webinar 2011 1 Evaluation Considerations: Measures & Methods Shrikant I. Bangdiwala, PhD Professor of Research in Biostatistics Injury Prevention.](https://reader035.fdocuments.net/reader035/viewer/2022062619/55153a625503465e608b5dbd/html5/thumbnails/58.jpg)
WHO VIP Webinar 201158
Systematic reviews
A protocol driven comprehensive review and synthesis of data focusing on a topic or on related key questions formulate specific key questions developing a protocol refining the questions of interest conducting a literature search for evidence selecting studies that meet the inclusion criteria appraising the studies critically synthesizing and interpreting the results
![Page 59: WHO VIP Webinar 2011 1 Evaluation Considerations: Measures & Methods Shrikant I. Bangdiwala, PhD Professor of Research in Biostatistics Injury Prevention.](https://reader035.fdocuments.net/reader035/viewer/2022062619/55153a625503465e608b5dbd/html5/thumbnails/59.jpg)
WHO VIP Webinar 201159
Example – Systematic review
Shults et al (2001) Amer J Prev Med
![Page 60: WHO VIP Webinar 2011 1 Evaluation Considerations: Measures & Methods Shrikant I. Bangdiwala, PhD Professor of Research in Biostatistics Injury Prevention.](https://reader035.fdocuments.net/reader035/viewer/2022062619/55153a625503465e608b5dbd/html5/thumbnails/60.jpg)
WHO VIP Webinar 201160
Systematic reviews
Of particular value in bringing together a number of separately conducted studies, sometimes with conflicting findings, and synthesizing their results.
To this end, systematic reviews may or may not include a statistical synthesis called meta-analysis, depending on whether the studies are similar enough so that combining their results is meaningful Green (2005) Singapore Medical Journal
Zaza et al (2001) Amer. J Preventive Medicine – motor vehicle
![Page 61: WHO VIP Webinar 2011 1 Evaluation Considerations: Measures & Methods Shrikant I. Bangdiwala, PhD Professor of Research in Biostatistics Injury Prevention.](https://reader035.fdocuments.net/reader035/viewer/2022062619/55153a625503465e608b5dbd/html5/thumbnails/61.jpg)
WHO VIP Webinar 201161
Meta analysis
A method of combining the results of studies quantitatively to obtain a summary estimate of the effect of an interventionOften restricted to randomized controlled trialsRecently, the Cochrane Collaboration is
‘branching out’ to include both experimental and observational studies in meta analyses
![Page 62: WHO VIP Webinar 2011 1 Evaluation Considerations: Measures & Methods Shrikant I. Bangdiwala, PhD Professor of Research in Biostatistics Injury Prevention.](https://reader035.fdocuments.net/reader035/viewer/2022062619/55153a625503465e608b5dbd/html5/thumbnails/62.jpg)
WHO VIP Webinar 201162
Meta analysis
e.g. Liu et al (2008) Cochrane Collaboration
![Page 63: WHO VIP Webinar 2011 1 Evaluation Considerations: Measures & Methods Shrikant I. Bangdiwala, PhD Professor of Research in Biostatistics Injury Prevention.](https://reader035.fdocuments.net/reader035/viewer/2022062619/55153a625503465e608b5dbd/html5/thumbnails/63.jpg)
WHO VIP Webinar 201163
Meta analysis
The combining of results should take into account: the ‘quality’ of the studies
• Assessed by the reciprocal of the variance the ‘heterogeneity’ among the studies
• Assessed by the variance between studies
![Page 64: WHO VIP Webinar 2011 1 Evaluation Considerations: Measures & Methods Shrikant I. Bangdiwala, PhD Professor of Research in Biostatistics Injury Prevention.](https://reader035.fdocuments.net/reader035/viewer/2022062619/55153a625503465e608b5dbd/html5/thumbnails/64.jpg)
WHO VIP Webinar 201164
Meta analysis – estimation of effect
The estimate is a weighted average, where the weight of a study is the reciprocal of its variance
In order to calculate the variance of a study, one can use either a ‘fixed’ effects model or a ‘mixed’/’random’ effects model Fixed effects model:
utilizes no information from other studies Random effects model:
considers variance among and within studies
*2
2
)var(
)var(
iYi
iii
VY
eY
Borenstein et al (2009) Introduction to Meta Analysis
iYii VeY )var()var(
![Page 65: WHO VIP Webinar 2011 1 Evaluation Considerations: Measures & Methods Shrikant I. Bangdiwala, PhD Professor of Research in Biostatistics Injury Prevention.](https://reader035.fdocuments.net/reader035/viewer/2022062619/55153a625503465e608b5dbd/html5/thumbnails/65.jpg)
WHO VIP Webinar 201165
Meta analysis & meta regression
Dealing with ‘heterogeneity’ among the studies - 2 Decompose the total variance into among and within
components using mixed effects models for getting a more precise estimate of the intervention effect
If there is still residual heterogeneityExpand the mixed effects model to include study-level
covariates that may explain some of the residual variability among studies meta regression
![Page 66: WHO VIP Webinar 2011 1 Evaluation Considerations: Measures & Methods Shrikant I. Bangdiwala, PhD Professor of Research in Biostatistics Injury Prevention.](https://reader035.fdocuments.net/reader035/viewer/2022062619/55153a625503465e608b5dbd/html5/thumbnails/66.jpg)
WHO VIP Webinar 201166
Meta regression
e.g.
iiiii eXXY 2211
Overall mean
X1 study variable – EU/USA
X2 study variable – population type
study random effect
random error
![Page 67: WHO VIP Webinar 2011 1 Evaluation Considerations: Measures & Methods Shrikant I. Bangdiwala, PhD Professor of Research in Biostatistics Injury Prevention.](https://reader035.fdocuments.net/reader035/viewer/2022062619/55153a625503465e608b5dbd/html5/thumbnails/67.jpg)
WHO VIP Webinar 201167
Meta analysis
Standard meta-analytical methods are typically restricted to comparisons of 2 interventions using direct, head-to-head evidence alone.
So, for example, if we are interested in the Intervention A vs Intervention B comparison, we would include only studies that compare Intervention A versus Intervention B directly.
Many times we have multiple types of interventions for the same type of problem, and we hardly have head-to-head comparisons
We may also have multiple component interventions
![Page 68: WHO VIP Webinar 2011 1 Evaluation Considerations: Measures & Methods Shrikant I. Bangdiwala, PhD Professor of Research in Biostatistics Injury Prevention.](https://reader035.fdocuments.net/reader035/viewer/2022062619/55153a625503465e608b5dbd/html5/thumbnails/68.jpg)
WHO VIP Webinar 201168
Mixed treatment meta analysis
Let the outcome variable be a binary response 1 = positive response 0 = negative response
We can calculate the binomial counts out of a total number at risk on the kth intervention in the jth study
We can then calculate
the estimated probability of the outcome (risk of response) for the kth intervention in the jth study
kj
kjkj n
rp
:
::
kjr :kjn :
Welton et al 2009 Amer J Epid
![Page 69: WHO VIP Webinar 2011 1 Evaluation Considerations: Measures & Methods Shrikant I. Bangdiwala, PhD Professor of Research in Biostatistics Injury Prevention.](https://reader035.fdocuments.net/reader035/viewer/2022062619/55153a625503465e608b5dbd/html5/thumbnails/69.jpg)
WHO VIP Webinar 201169
Mixed treatment meta analysis
Let each study have a reference ‘‘standard’’ intervention arm, sj, with study-specific ‘‘standard’’ log odds of outcome, j .
The log odds ratio, j:k, of outcome for intervention k, relative to standard sj, is assumed to come from a random effects model with mean log odds ratio , and between-study standard deviation
where dk is the mean log odds ratio of outcome for intervention k relative to control (so that d1 = 0).
)(jsk dd
Welton et al 2009 Amer J Epid
![Page 70: WHO VIP Webinar 2011 1 Evaluation Considerations: Measures & Methods Shrikant I. Bangdiwala, PhD Professor of Research in Biostatistics Injury Prevention.](https://reader035.fdocuments.net/reader035/viewer/2022062619/55153a625503465e608b5dbd/html5/thumbnails/70.jpg)
WHO VIP Webinar 201170
Mixed treatment meta analysis
This leads to the following logistic regression model:
where ]),[(~: jskkj ddN
k
s
p
p
kjj
jj
kj
kj
int
int
1ln
::
:
Welton et al 2009 Amer J Epid
![Page 71: WHO VIP Webinar 2011 1 Evaluation Considerations: Measures & Methods Shrikant I. Bangdiwala, PhD Professor of Research in Biostatistics Injury Prevention.](https://reader035.fdocuments.net/reader035/viewer/2022062619/55153a625503465e608b5dbd/html5/thumbnails/71.jpg)
WHO VIP Webinar 201171
Mixed treatment meta analysis- multiple-methods interventions
If we have multiple methods in the ith intervention
Plus we have multiple times when the outcome is assessed
itiititiit eXMMtY 423121
,...,, 321 iii MMM
Study effect
Time effect
Components 1 & 2 effects
Study covariable
Error term
![Page 72: WHO VIP Webinar 2011 1 Evaluation Considerations: Measures & Methods Shrikant I. Bangdiwala, PhD Professor of Research in Biostatistics Injury Prevention.](https://reader035.fdocuments.net/reader035/viewer/2022062619/55153a625503465e608b5dbd/html5/thumbnails/72.jpg)
WHO VIP Webinar 201172
Statistical analysis
Methodology does exist for developing stronger collective evidence, evaluating the effectiveness of community based interventions, using different types of study designs and interventions
Developing “practice-based evidence”
![Page 73: WHO VIP Webinar 2011 1 Evaluation Considerations: Measures & Methods Shrikant I. Bangdiwala, PhD Professor of Research in Biostatistics Injury Prevention.](https://reader035.fdocuments.net/reader035/viewer/2022062619/55153a625503465e608b5dbd/html5/thumbnails/73.jpg)
Safety 2010 London73
Dissemination
We should not stop at developing the evidence We must work alongside economists in developing
ways to effectively communicate ‘what works’ methodology and cost models do exist for estimating the “return on investment”
Money talks !!
WHO VIP Webinar 201173
![Page 74: WHO VIP Webinar 2011 1 Evaluation Considerations: Measures & Methods Shrikant I. Bangdiwala, PhD Professor of Research in Biostatistics Injury Prevention.](https://reader035.fdocuments.net/reader035/viewer/2022062619/55153a625503465e608b5dbd/html5/thumbnails/74.jpg)
WHO VIP Webinar 201174
Challenges – Evaluation requires
Integration of evaluation from the beginning Appropriate measures, possible to be collected objectively,
unbiasedly, easily and with completeness Appropriate qualitative and process information, to complement
the quantitative information Concrete and convincing evidence of what aspects work in
individual communities Formal methodological statistical evaluation of specific elements
of programs Collective evidence of what common elements of programs work Effective dissemination strategies – “return on investment”
![Page 75: WHO VIP Webinar 2011 1 Evaluation Considerations: Measures & Methods Shrikant I. Bangdiwala, PhD Professor of Research in Biostatistics Injury Prevention.](https://reader035.fdocuments.net/reader035/viewer/2022062619/55153a625503465e608b5dbd/html5/thumbnails/75.jpg)
WHO VIP Webinar 201175 75