UNPACKING THE BLACK BOXENGINEERING MORE POTENT BEHAVIORALINTERVENTIONS TO IMPROVE PUBLIC HEALTH
Linda M. Collins, Ph.D.
Outline What is a behavioral intervention? What is the black box, and why unpack it? What is optimization? A few examples Concluding remarks
What is a behavioral intervention?
Definition: A program aimed at modifying behavior for the purpose of preventing/treating disease, promoting health, and/or enhancing well-being.
What is a behavioral intervention?
A few examples from PSU (out of many):
PROSPER (Greenberg, Bierman, Feinberg, Welsh, Perkins, Mincemoyer, Corbin)
Keepin’ it REAL (Hecht, Miller-Day, Graham)
Project ACT (Turrisi)
Active MOMS (Downs)
Interventions for at-risk caregivers (Zarit)
Head Start REDI (Bierman, Domitrovich)
HealthWise (Caldwell, Smith)
RESERVE (Kolanowski, Fick)
What is a behavioral intervention?
Most behavioral interventions are made up of multiple components.
Some components may be pharmaceutical or medical.
Outline What is a behavioral intervention? What is the black box, and why unpack it? What is optimization? A few examples Concluding remarks
Treatment package approach
component
component
component
RCT
component component
What is the black box, and why unpack it?
If RCT finds significant effect, it is UNKNOWN Which components are making positive
contribution to overall effect Whether all the components are really needed Whether a component’s contribution offsets its
cost How to make the intervention more effective,
efficient, cost-effective
What is the black box, and why unpack it?
If RCT finds non-significant effect, it is UNKNOWN Whether any components in the box are worth
retaining
Whether one component in the box had a negative effect that offset the positive effect of others
Specifically what went wrong and how to do it better the next time
What is the black box, and why unpack it?
The treatment package approach Encourages stuffing the black box with as
many components as possible to get a significant effect
Downplays considerations such as efficiency, cost-effectiveness, time-effectiveness
Places focus on attaining statistical significance rather than meeting a criterion
What is the black box, and why unpack it?
This is NOT how engineers build products. They take an approach that is
► Systematic
► Efficient
► Focused on the clear objective of optimizing the product
Can we borrow ideas from engineering… … and build optimized behavioral interventions?
Resource management principle How engineers think: This is what I need to find out: ______ These are the resources I have: ______ How can I manage my resources strategically
to find out what I need to know?
Resource management principle Logic: Objective is to identify ONE OF THE
TWO OR THREE BEST approaches
0
1
2
3
4
5
6
7
8
9
10
Hypothetical distribution of outcome variable for different combinations of components
Score on outcome variable
Freq
uenc
y
Resource management principle Logic: Objective is to identify ONE OF THE
TWO OR THREE BEST approaches Manage research resources strategically
► Decide what information most important, and target resources there
Choose efficient experimental designs Take calculated risks
Resource management principle Note that the starting point is the resources
you have By definition, MOST does not require an
increase in research resources► But may require a realignment of research
resources
The Multiphase Optimization Strategy (MOST)
component
component
component
Optimizedintervention
component
component
component
RCT
component component
Screening (component
selection) and refinement
experiment(s)
Note: MOST is a framework, not an off-the-shelf procedure.
Outline What is a behavioral intervention? What is the black box, and why unpack it? What is optimization? A few examples Concluding remarks
Definition of optimized “The best possible solution… subject to given
constraints” [emphasis added] (The Concise Oxford Dictionary of Mathematics)
► Optimized does not mean best in an absolute or ideal sense
► Instead, realistic because it includes constraints
Optimization always involves a clearly stated optimization criterion
► A working definition of what YOU mean by “better”
Selecting an optimization criterion: what you mean by “best”
Your definition of “best possible, given constraints”
This is the goal you want to achieve Suppose you are developing a behavioral
intervention to encourage HAART adherence in HIV+ people called “Living with HIV”
One possible optimization criterion: Intervention with no “dead wood”
► Example: Health care settings are finding it difficult to fit Living with HIV into their busy day, and are watching costs carefully. The investigators want to be confident that every component is necessary so that no time or money is wasted.
► Achieve this by selecting only active intervention components.
Another possible optimization criterion:
Most effective intervention that can be delivered for ≤ some $$
► Example: To have a realistic chance of being adopted by HMOs, Living with HIV must cost no more than $200/participant to deliver, including materials and staff time.
► Achieve this by selecting set of components that represents the most effective intervention that can be delivered for ≤ $200.
Another possible optimization criterion:
Most effective intervention that can be delivered in ≤ some amount of time
► Example: Interviews with health care clinic staff suggest that Living with HIV has the best chance of being implemented well if it takes no more than 15 minutes to deliver.
► Achieve this by selecting set of components that represents the most effective intervention that can be delivered in ≤ 15 minutes.
Evaluation and optimization:Both important;
not the same thing.
Evaluation: Is the intervention’s effect
statistically significant?Optimization: Is the intervention the best possible, given constraints?
No Yes
NoMay wish to
optimize using effect size as
criterion
Intervention can probably be
improved
Yes Different intervention
strategy neededWhat we should
be aiming for
Outline What is a behavioral intervention? What is the black box, and why unpack it? What is optimization? A few examples Concluding remarks
Three examples of MOST All currently in the field All three have the same optimization criterion:
No inactive components
Example 1: Clinic-based smoking cessation study funded by NCI
Tim Baker Mike Fiore University of Wisconsin
Team also includes B.A. Christiansen, L.M. Collins, J.W. Cook, D.E. Jorenby, R.J. Mermelstein, M.E. Piper, T.R. Schlam, S.S. Smith
Project funded by NCI grant P50CA143188
Example 1: Clinic-based smoking cessation study funded by NCI
Tim Baker Mike Fiore University of Wisconsin
Objective: Develop an effective “lean” clinic-based smoking cessation intervention (no inactive
components)
Some interesting features of Example 1
Study being implemented in health care settings
Involves both behavioral and pharmaceutical components
Baker & Fiore’s model of the smoking cessation process
PRECESSATION(3 weeks prior up to
quit day)
CESSATION(quit day to 2 weeks
after quit day)
MAINTENANCE(2 weeks to 6 months
after quit day)MOTIVATION
Six components being considered for the smoking cessation intervention
Precessation nicotine patch (No, Yes) Precessation ad lib nicotine gum (No, Yes) Precessation in-person counseling (No, Yes) Cessation in-person counseling (Minimal,
Intensive) Cessation phone counseling (Minimal, Intensive) Maintenance medication duration (Short, Long)
Experiment to examine individual component effects
We decided to conduct a factorial experiment► Special type called a fractional factorial
N=512 subjects TOTAL provides power ≥ .8
Engineering the intervention Experiment will give us
► Main effect of each individual intervention component on outcomes► e.g. number days abstinent in 2-wk post-quit period
► Selected interactions between intervention components
This information will be used to select components/component levels
Result: optimized clinic-based smoking cessation intervention
Plan to conduct an RCT to establish statistical significance of effect
Example 2: School-based drug abuse/HIV prevention study funded by NIDA
Linda L. Caldwell Edward A. SmithPenn State
Team also includes D. Coffman, L. Collins, J. Cox, I. Evans, J. Graham, M. Greenberg, J. Jacobs, D. Jones, M. Lai, C. Matthews, R. Spoth, L. Wegner, T. Vergnani, E. Weybright
Project funded by NIDA grant R01DA029084
Example 2: School-based drug abuse/HIV prevention study funded by NIDA
Linda L. Caldwell Edward A. SmithPenn State
Objective: To develop a strategy for maintaining implementation fidelity in which all components contribute
Some interesting features of Example 2
Components being examined relate to how the intervention is delivered
► “sealed intervention”
Cluster randomization
Background HealthWise school-based ATOD/HIV prevention
intervention Has previously been evaluated in South Africa Metropolitan South Education District in South
Africa wants to implement HealthWise in all its schools
Question: how to maintain fidelity? Metro South allowed us to conduct an experiment
Caldwell & Smith’s model of implementation fidelity
Components Enhanced teacher training
► Standard training (one and one-half days) vs. enhanced (three days + two additional days four months later)
Structure, support, and supervision► No additional vs. additional (e.g., weekly text messages; monthly
visits from support staff; option to call support staff with questions as needed)
Enhanced school climate► No climate enhancement vs. climate enhancement (e.g., form
committee of parents and teachers to promote HealthWise; develop visuals; issue newsletter)
How to conduct an experiment to examine individual component
effects We decided to conduct a factorial experiment. Why? Enables examination of individual component effects
AND Statistical power achieved with smaller sample sizes
than alternative designs► Yes, I mean smaller
BUT they also usually require more experimental conditions than we may be accustomed to
Experiment uses all 56 schools in district
Factorial experiments 101 Example: 2 X 2, or 22, factorial design
Factorial experiments can have► ≥ 2 factors
► ≥ 2 levels per factor
Component A
Component B Off On
Off A,B off A on, B off
On A off, B on A,B on
Experi-mental
conditionN of
schoolsHealthWise
program Training
Structure, support, & supervision
Enhanced school climate
1 7 2 7 3 7 4 7 5 7 6 7 7 7 8 7
HealthWise experiment in South African school district. 56 schools in all; 7 schools assigned to each experimental condition
Are 7 schools per experimental condition enough?
We estimated power ≥ .8 for main effects Remember that each main effect estimate is
based on ALL schools In a factorial experiment you DO NOT
compare individual conditions
Experi-mental
conditionN of
schoolsHealthWise
program Training
Structure, support, & supervision
Enhanced school climate
1 7 2 7 3 7 4 7 5 7 6 7 7 7 8 7
Main effect of Training is mean of (5,6,7,8) vs. mean of (1,2,3,4).
Note that all 56 schools are used in estimating the main effect.
Main effect of Structure, support, & supervision is mean of (3,4,7,8) vs. mean of (1,2,5,6).
Note that all 56 schools are used in estimating the main effect.
Experi-mental
conditionN of
schoolsHealthWise
program Training
Structure, support, & supervision
Enhanced school climate
1 7 2 7 3 7 4 7 5 7 6 7 7 7 8 7
Experi-mental
conditionN of
schoolsHealthWise
program Training
Structure, support, & supervision
Enhanced school climate
1 7 2 7 3 7 4 7 5 7 6 7 7 7 8 7
Main effect of Enhanced school climate is mean of (2.4.6.8) vs. mean of (1,3,5,7).
Note that all 56 schools are used in estimating the main effect.
Engineering the intervention Experiment will give us
► Main effect of each individual intervention component on outcome variables
► Also interactions between intervention components
This information will be used to select the best set of the three components
Result: Intervention engineered to optimize fidelity according to our criterion
Note: optimized ≠ the best possible
Example 3: Internet-delivered drug abuse prevention program aimed at NCAA
athletes
David Wyrick UNC Greensboro
Prevention Strategies, LLC
Melodie Fearnow-KenneyPrevention Strategies, LLC
Team includes L. M. Collins, J. Milroy, K. RulisonProject funded by NIDA grant R44DA023735
Example 3: Internet-delivered drug abuse prevention program aimed at NCAA
athletes
OBJECTIVE: No inactive components. Secondary aim: Maximize overall program effect within available research
resources.
David Wyrick UNC Greensboro
Prevention Strategies, LLC
Melodie Fearnow-KenneyPrevention Strategies, LLC
Some interesting features of Example 3
Intervention is internet-delivered Experimental design is a cluster-randomized
fractional factorial Takes an iterative approach
Background of MyPlaybook Study
Theoretical Model for MyPlaybook
Social NormsSocial Norms Theory
Positive & Negative Expectancies
Health Belief Model
Intentions to Avoid Use and Prevent Harm
Theory of Planned Behavior
Alcohol and Other Drug Use
Consequences
Intervention components Five components, each corresponding to a substance:
I. Alcohol
II. Tobacco
III. Marijuana
IV. Performance enhancers
V. Prescription and OTC drugs Each component aimed at
► social norms
► positive and negative expectancies
► intentions to avoid use and prevent harm
MOST as implemented in the My Playbook study
Optimizedintervention
Alcohol
Performance enhancers
Tobacco
RCT
Marijuana
Prescription/OTC drugs
Experiment 1
Revision of componentsExperiment 2
Design of MyPlaybook experiments: 25-1 fractional factorial. 64 schools assigned to conditions; overall N approx. 3500
Condition Number
Intro Component
Intervention Components Manipulated
I II III IV V
1 On Off Off Off Off On
2 On Off Off Off On Off
3 On Off Off On Off Off
4 On Off Off On On On
5 On Off On Off Off Off
6 On Off On Off On On
7 On Off On On Off On
8 On Off On On On Off
9 On On Off Off Off Off
10 On On Off Off On On
11 On On Off On Off On
12 On On Off On On Off
13 On On On Off Off On
14 On On On Off On On
15 On On On On Off Off
16 On On On On On On
Engineering the intervention Each experiment will give us
► Main effect of each individual intervention component on outcome variables
► e.g. substance use at 30-day follow-up
► Also selected interactions between intervention components
We will select the best version of each component for inclusion in the optimized intervention
The optimized intervention will be evaluated via an RCT
Outline What is a behavioral intervention? What is the black box, and why unpack it? What is optimization? A few examples Concluding remarks
Frequently asked questions Nice idea, but will it be fundable? Can this approach be carried out with the level of
funding typically available? Will I be able to publish based on the results of the
screening experiments? I don’t see how I can implement all the experimental
conditions required by a factorial experiment. What if the outcome of interest is in the distant
future?
If you are interested in learning more about MOST
FACULTY► “A Taste of Methodology” brief workshop, Monday,
April 30, 10:30 am to 1:30 pm (co-sponsored by the Methodology Center and SSRI; includes lunch)
GRAD STUDENTS► HDFS 597x Quantitative Methods for Intervention
Science
The future I would like to see Behavioral interventions systematically
engineered to meet specific criteria Standards of intervention effectiveness,
efficiency, cost-effectiveness Cumulative base of scientific knowledge Behavioral intervention engineering a subfield
in industrial engineering
THANK YOU!
Top Related