© 2012 | WORK, FAMILY & HEALTH NETWORK Evaluating Workplace Health and Wellbeing Interventions...
-
Upload
georgia-parsons -
Category
Documents
-
view
215 -
download
0
Transcript of © 2012 | WORK, FAMILY & HEALTH NETWORK Evaluating Workplace Health and Wellbeing Interventions...
![Page 1: © 2012 | WORK, FAMILY & HEALTH NETWORK Evaluating Workplace Health and Wellbeing Interventions Georgia T. Karuntzos, Ph.D Jeremy Bray, Ph.D Jesse M. Hinde.](https://reader035.fdocuments.net/reader035/viewer/2022081517/5697bfb61a28abf838c9e4ea/html5/thumbnails/1.jpg)
© 2012 | WORK, FAMILY & HEALTH NETWORK
Evaluating Workplace Health and Wellbeing Interventions
Georgia T. Karuntzos, Ph.D
Jeremy Bray, Ph.D
Jesse M. Hinde
RTI International
![Page 2: © 2012 | WORK, FAMILY & HEALTH NETWORK Evaluating Workplace Health and Wellbeing Interventions Georgia T. Karuntzos, Ph.D Jeremy Bray, Ph.D Jesse M. Hinde.](https://reader035.fdocuments.net/reader035/viewer/2022081517/5697bfb61a28abf838c9e4ea/html5/thumbnails/2.jpg)
Steps for Conducting Effective Evaluations
2
Centers for Disease Control
Standards
UtilityFeasibilityProprietyAccuracy
Step 1: Identify Need and
Engage Stakeholders
Step 2: Describe the
Program and it’s intended effects (Logic Model)
Step 4: Gather credible
evidence
Step 3: Focus theEvaluation
design
Step 5:Justify
conclusions
Step 6:Ensure useand share
lessons learned
![Page 3: © 2012 | WORK, FAMILY & HEALTH NETWORK Evaluating Workplace Health and Wellbeing Interventions Georgia T. Karuntzos, Ph.D Jeremy Bray, Ph.D Jesse M. Hinde.](https://reader035.fdocuments.net/reader035/viewer/2022081517/5697bfb61a28abf838c9e4ea/html5/thumbnails/3.jpg)
Types of Evaluations
Adapted from:Norland, E. (2004, Sept). From education theory.. to conservation practice Presented at the Annual Meeting of the International Association for Fish & Wildlife Agencies, Atlantic City, New Jersey.
![Page 4: © 2012 | WORK, FAMILY & HEALTH NETWORK Evaluating Workplace Health and Wellbeing Interventions Georgia T. Karuntzos, Ph.D Jeremy Bray, Ph.D Jesse M. Hinde.](https://reader035.fdocuments.net/reader035/viewer/2022081517/5697bfb61a28abf838c9e4ea/html5/thumbnails/4.jpg)
Theoretical/Logic Model
■ A diagram of the theory of how a program is supposed to work—a graphic depiction of relationships between activities and results (conceptual/theoretical model)
■ A logical chain of connections showing what a program intends to accomplish
■ Increases intentionality and purpose
■ Guides prioritization and allocation of resources
■ Helps to identify important variables to measure; use evaluation resources wisely
■ Supports replication
![Page 5: © 2012 | WORK, FAMILY & HEALTH NETWORK Evaluating Workplace Health and Wellbeing Interventions Georgia T. Karuntzos, Ph.D Jeremy Bray, Ph.D Jesse M. Hinde.](https://reader035.fdocuments.net/reader035/viewer/2022081517/5697bfb61a28abf838c9e4ea/html5/thumbnails/5.jpg)
![Page 6: © 2012 | WORK, FAMILY & HEALTH NETWORK Evaluating Workplace Health and Wellbeing Interventions Georgia T. Karuntzos, Ph.D Jeremy Bray, Ph.D Jesse M. Hinde.](https://reader035.fdocuments.net/reader035/viewer/2022081517/5697bfb61a28abf838c9e4ea/html5/thumbnails/6.jpg)
WFHN Evaluation Design
■ Single protocol■ Work redesign, supervisor training and self-monitoring
■ Multiple industries and worksites■ Healthcare (blue collar) ■ Telecommunications (white collar)
■ Group randomized field experiment■ Intervention and comparison group assignment at the
worksite or work group level■ Adaptive randomization to balance covariates
![Page 7: © 2012 | WORK, FAMILY & HEALTH NETWORK Evaluating Workplace Health and Wellbeing Interventions Georgia T. Karuntzos, Ph.D Jeremy Bray, Ph.D Jesse M. Hinde.](https://reader035.fdocuments.net/reader035/viewer/2022081517/5697bfb61a28abf838c9e4ea/html5/thumbnails/7.jpg)
Outcome Evaluation Design
■ Nested cohort design
■ Worksite partners are randomized within industry to intervention or control group
■ Outcomes evaluated at multiple levels, employees, workgroups and worksites
■ Allows for multiple levels of clustering (e.g., workgroups within worksites), and a variety of outcomes (e.g., discrete, continuous, count)
![Page 8: © 2012 | WORK, FAMILY & HEALTH NETWORK Evaluating Workplace Health and Wellbeing Interventions Georgia T. Karuntzos, Ph.D Jeremy Bray, Ph.D Jesse M. Hinde.](https://reader035.fdocuments.net/reader035/viewer/2022081517/5697bfb61a28abf838c9e4ea/html5/thumbnails/8.jpg)
Formative(Process) Evaluation Framework
8
Implementation Outcomes
Acceptability Adoption
AppropriatenessCost
FeasibilityFidelity
PenetrationSustainability
-
*IOM Standards of Care
What?
QIsESTs
How?
ImplementationStrategies
Implementation Research Methods
ServiceOutcomes*Efficiency
SafetyEffectiveness
EquityPatient-
centerednessTimeliness
Individual Outcomes
SatisfactionFunction
Health status/symptoms
Proctor’s Model of Implementation Outcomes
![Page 9: © 2012 | WORK, FAMILY & HEALTH NETWORK Evaluating Workplace Health and Wellbeing Interventions Georgia T. Karuntzos, Ph.D Jeremy Bray, Ph.D Jesse M. Hinde.](https://reader035.fdocuments.net/reader035/viewer/2022081517/5697bfb61a28abf838c9e4ea/html5/thumbnails/9.jpg)
Process Evaluation Data■ Document Reviews – provide information to build an “a priori”
understanding of the program content, operations, context, and program stakeholders
■ Review reports, instruments, protocols, promotional materials, patient materials, resource lists, organizational documents (org charts, flow charts, operation manuals), web sites.
■ Observational Studies – provide empirical evidence to assess program fidelity, and generate service flow and timing data to inform outcome and cost analysis
■ Key Informant Interviews – provide contextual information related to program utility, contextual factors that influence program implementation, service delivery, dispersion, and sustainability
■ Practioner and Consumer Surveys -- provide systematic data related to service delivery experiences and program related perceptions
![Page 10: © 2012 | WORK, FAMILY & HEALTH NETWORK Evaluating Workplace Health and Wellbeing Interventions Georgia T. Karuntzos, Ph.D Jeremy Bray, Ph.D Jesse M. Hinde.](https://reader035.fdocuments.net/reader035/viewer/2022081517/5697bfb61a28abf838c9e4ea/html5/thumbnails/10.jpg)
Qualitative Analysis Methods
■ Recursive abstraction (Document Summaries)
■ Iterative process that generates summaries, classifications, lists, rates, or groupings
■ Deductive and Inductive Analysis (data coding)
■ Deductive “a priori” framework
■ Inductive “grounded theory” analysis
■ Results in taxonomies, themes, categories, orders
■ Comparative analysis
■ Documented vs observed processes, behaviors or outputs
■ Document changes over time (e.g., model migration)
■ Variation between processes or outputs
■ Mixed Methods (triangulation and convergence)10
![Page 11: © 2012 | WORK, FAMILY & HEALTH NETWORK Evaluating Workplace Health and Wellbeing Interventions Georgia T. Karuntzos, Ph.D Jeremy Bray, Ph.D Jesse M. Hinde.](https://reader035.fdocuments.net/reader035/viewer/2022081517/5697bfb61a28abf838c9e4ea/html5/thumbnails/11.jpg)
Process Evaluation Results
Comprehensive description of program components
Performance indicators and proficiency scores for program delivery
Common barriers and facilitators across worksites
Descriptive taxonomy of program settings
Construction of moderator variables for use in outcome and economic analyses
Program delivery protocols that are used at each worksite site
![Page 12: © 2012 | WORK, FAMILY & HEALTH NETWORK Evaluating Workplace Health and Wellbeing Interventions Georgia T. Karuntzos, Ph.D Jeremy Bray, Ph.D Jesse M. Hinde.](https://reader035.fdocuments.net/reader035/viewer/2022081517/5697bfb61a28abf838c9e4ea/html5/thumbnails/12.jpg)
Outcome Evaluation Questions■ Program Outputs
■ To what extent is the program actually performed as measured by (e.g. the number of health risk appraisals completed, percentage of employees participating in workshops, number of follow-up services delivered)
■ Proximal and Distal Outcomes
■ What is the effectiveness of the program on outcomes of interest? (changes in sleep quality, changes in bio-measures)?
■ What stakeholder and employee-level characteristics moderate the willingness and ability of performance sites and practitioners to adopt the program
![Page 13: © 2012 | WORK, FAMILY & HEALTH NETWORK Evaluating Workplace Health and Wellbeing Interventions Georgia T. Karuntzos, Ph.D Jeremy Bray, Ph.D Jesse M. Hinde.](https://reader035.fdocuments.net/reader035/viewer/2022081517/5697bfb61a28abf838c9e4ea/html5/thumbnails/13.jpg)
WFHN Study Data
![Page 14: © 2012 | WORK, FAMILY & HEALTH NETWORK Evaluating Workplace Health and Wellbeing Interventions Georgia T. Karuntzos, Ph.D Jeremy Bray, Ph.D Jesse M. Hinde.](https://reader035.fdocuments.net/reader035/viewer/2022081517/5697bfb61a28abf838c9e4ea/html5/thumbnails/14.jpg)
Outcome Evaluation Methods
■ Descriptive statistics generating classifications or groupings of the participating worksites, provider characteristics, employee characteristics
■ Multivariate regression examining the relationship between groups
■ Statistical modeling measuring changes on outcomes over time
![Page 15: © 2012 | WORK, FAMILY & HEALTH NETWORK Evaluating Workplace Health and Wellbeing Interventions Georgia T. Karuntzos, Ph.D Jeremy Bray, Ph.D Jesse M. Hinde.](https://reader035.fdocuments.net/reader035/viewer/2022081517/5697bfb61a28abf838c9e4ea/html5/thumbnails/15.jpg)
Outcome Evaluation Results
■ Rates and frequencies of program participation
■ Performance site descriptive characteristics (including staff characteristics) that are associated with outcomes of interest
■ Within group pre to post changes in health and wellness outcomes
■ Between group comparative changes in health and wellness outcomes at multiple time points
![Page 16: © 2012 | WORK, FAMILY & HEALTH NETWORK Evaluating Workplace Health and Wellbeing Interventions Georgia T. Karuntzos, Ph.D Jeremy Bray, Ph.D Jesse M. Hinde.](https://reader035.fdocuments.net/reader035/viewer/2022081517/5697bfb61a28abf838c9e4ea/html5/thumbnails/16.jpg)
![Page 17: © 2012 | WORK, FAMILY & HEALTH NETWORK Evaluating Workplace Health and Wellbeing Interventions Georgia T. Karuntzos, Ph.D Jeremy Bray, Ph.D Jesse M. Hinde.](https://reader035.fdocuments.net/reader035/viewer/2022081517/5697bfb61a28abf838c9e4ea/html5/thumbnails/17.jpg)
Intervention Effect on Work Environment
FSWE Hypotheses
■H1. There are changes in work environment for the
■ H1a. TOMO intervention group
■ H1b. TOMO control group
■ H1c. LEEF intervention group
■ H1d. LEEF control group
■H2. There are baseline differences in environment between intervention and control groups for the
■ H2a. TOMO industry
■ H2b. LEEF industry
■H3. There are differences in the rate of change in climate between intervention and control groups for the
■ H3a. TOMO industry
■ H3b. LEEF industry
![Page 18: © 2012 | WORK, FAMILY & HEALTH NETWORK Evaluating Workplace Health and Wellbeing Interventions Georgia T. Karuntzos, Ph.D Jeremy Bray, Ph.D Jesse M. Hinde.](https://reader035.fdocuments.net/reader035/viewer/2022081517/5697bfb61a28abf838c9e4ea/html5/thumbnails/18.jpg)
![Page 19: © 2012 | WORK, FAMILY & HEALTH NETWORK Evaluating Workplace Health and Wellbeing Interventions Georgia T. Karuntzos, Ph.D Jeremy Bray, Ph.D Jesse M. Hinde.](https://reader035.fdocuments.net/reader035/viewer/2022081517/5697bfb61a28abf838c9e4ea/html5/thumbnails/19.jpg)
FSWE Factor Analysis
Family Supportive Work Environment
Family Supportive Supervisor Behaviors
Organizational Climate Time Adequacy Schedule Control/
Decision Authority
0
11.4 8.3 10.7 11.5
1.465 1.142 0.655 0.844
Family Supportive Work Environment-9.9
Family Supportive Supervisor Behaviors
Organizational Climate Time Adequacy Schedule Control/
Decision Authority
1.465 1.142 0.655 0.844
11.4 8.3 10.7 11.5
![Page 20: © 2012 | WORK, FAMILY & HEALTH NETWORK Evaluating Workplace Health and Wellbeing Interventions Georgia T. Karuntzos, Ph.D Jeremy Bray, Ph.D Jesse M. Hinde.](https://reader035.fdocuments.net/reader035/viewer/2022081517/5697bfb61a28abf838c9e4ea/html5/thumbnails/20.jpg)
Growth Model Results by Industry
Hypothesis Industry Group Random Effect
Estimate Standard Error p
H1a TOMO Intervention Slope 0.042 0.009 <0.0001
H1b TOMO Control Slope 0.014 0.008 0.081
H1c LEEF Intervention Slope 0.003 0.007 0.657
H1d LEEF Control Slope 0.001 0.006 0.905
H2a TOMO InterventionVs. Control
Intercept 0.033 0.038 0.393
H2b LEEF InterventionVs. Control
Intercept 0.013 0.027 0.619
H3a TOMO InterventionVs. Control
Slope 0.028 0.012 0.016
H3b LEEF InterventionVs. Control
Slope 0.002 0.009 0.804
![Page 21: © 2012 | WORK, FAMILY & HEALTH NETWORK Evaluating Workplace Health and Wellbeing Interventions Georgia T. Karuntzos, Ph.D Jeremy Bray, Ph.D Jesse M. Hinde.](https://reader035.fdocuments.net/reader035/viewer/2022081517/5697bfb61a28abf838c9e4ea/html5/thumbnails/21.jpg)
Economic Evaluation Questions
■ What is the program cost to worksites and to other stakeholders?
■ What is the cost-effectiveness of the program?
■ What characteristics moderate the cost-effectiveness of the program?
![Page 22: © 2012 | WORK, FAMILY & HEALTH NETWORK Evaluating Workplace Health and Wellbeing Interventions Georgia T. Karuntzos, Ph.D Jeremy Bray, Ph.D Jesse M. Hinde.](https://reader035.fdocuments.net/reader035/viewer/2022081517/5697bfb61a28abf838c9e4ea/html5/thumbnails/22.jpg)
Translational Study
An effective intervention is only useful if it is communicated to and adopted by workplaces
Complementary methods: Use participant feedback to inform
post-study messaging Analyze process data to study
employee perception for developing portrayals of the intervention
Assess potential dissemination channels
Stimulate market demand for effective intervention
![Page 23: © 2012 | WORK, FAMILY & HEALTH NETWORK Evaluating Workplace Health and Wellbeing Interventions Georgia T. Karuntzos, Ph.D Jeremy Bray, Ph.D Jesse M. Hinde.](https://reader035.fdocuments.net/reader035/viewer/2022081517/5697bfb61a28abf838c9e4ea/html5/thumbnails/23.jpg)
Thank You
For more information on the Work, Family, and Health Network Study
http://projects.iq.harvard.edu/wfhn/people