Experimental, Quasi- Experimental, and Ex Post Facto Designs
Experimental Designs for Optimizing InterventionsInbal (Billie) Nahum-Shani Experimental Designs for...
Transcript of Experimental Designs for Optimizing InterventionsInbal (Billie) Nahum-Shani Experimental Designs for...
![Page 1: Experimental Designs for Optimizing InterventionsInbal (Billie) Nahum-Shani Experimental Designs for Optimizing Interventions mHealth Training Institute August 2016. 1 Key Definition](https://reader033.fdocuments.net/reader033/viewer/2022053106/606fe9d3c5c8310527717ef3/html5/thumbnails/1.jpg)
NIH mHealth Institute Event
Inbal (Billie) Nahum-Shani
Experimental Designs for Optimizing Interventions
mHealth Training Institute August 2016
![Page 2: Experimental Designs for Optimizing InterventionsInbal (Billie) Nahum-Shani Experimental Designs for Optimizing Interventions mHealth Training Institute August 2016. 1 Key Definition](https://reader033.fdocuments.net/reader033/viewer/2022053106/606fe9d3c5c8310527717ef3/html5/thumbnails/2.jpg)
1
Key Definition Multi-Component Interventions
• Component: ► The content of the intervention (e.g., topics in prevention program)► The intervention modality (e.g., phone calls/emails) ► Features to promote compliance or adherence (e.g., reminder emails)
Example: • Optimizing a technology supported intervention for weight loss:
Bonnie Spring, PI. DK097364 ► Telephone Caching ► Report to Primary Care Provider ► Text Messages ► Meal Replacements► Buddy Training
![Page 3: Experimental Designs for Optimizing InterventionsInbal (Billie) Nahum-Shani Experimental Designs for Optimizing Interventions mHealth Training Institute August 2016. 1 Key Definition](https://reader033.fdocuments.net/reader033/viewer/2022053106/606fe9d3c5c8310527717ef3/html5/thumbnails/3.jpg)
2
How do We Typically Develop Interventions?
1. Scientific Model
2. Intervention Components
4. Confirm Effectiveness
3. Intervention Package
![Page 4: Experimental Designs for Optimizing InterventionsInbal (Billie) Nahum-Shani Experimental Designs for Optimizing Interventions mHealth Training Institute August 2016. 1 Key Definition](https://reader033.fdocuments.net/reader033/viewer/2022053106/606fe9d3c5c8310527717ef3/html5/thumbnails/4.jpg)
3
How do We Typically Develop Interventions?
1. Theoretical Model
2. Intervention Components
4. Confirm Effectiveness
3. Intervention Package
![Page 5: Experimental Designs for Optimizing InterventionsInbal (Billie) Nahum-Shani Experimental Designs for Optimizing Interventions mHealth Training Institute August 2016. 1 Key Definition](https://reader033.fdocuments.net/reader033/viewer/2022053106/606fe9d3c5c8310527717ef3/html5/thumbnails/5.jpg)
4
Open Questions Efficacy of Individual components
– Which components are effective?– Which level is more appropriate?– Which components work well together?
Sequencing of components– Which component to offer first?– Which to offer subsequently?– How should I tailor components over time?
![Page 6: Experimental Designs for Optimizing InterventionsInbal (Billie) Nahum-Shani Experimental Designs for Optimizing Interventions mHealth Training Institute August 2016. 1 Key Definition](https://reader033.fdocuments.net/reader033/viewer/2022053106/606fe9d3c5c8310527717ef3/html5/thumbnails/6.jpg)
5
Open Questions Efficacy of Individual components
– Which components are effective?– Which level is more appropriate?– Which components work well together?
Sequencing of components– Which component to offer first?– Which to offer subsequently?– How should I tailor components over time?
Factorial Designs
SMART
![Page 7: Experimental Designs for Optimizing InterventionsInbal (Billie) Nahum-Shani Experimental Designs for Optimizing Interventions mHealth Training Institute August 2016. 1 Key Definition](https://reader033.fdocuments.net/reader033/viewer/2022053106/606fe9d3c5c8310527717ef3/html5/thumbnails/7.jpg)
6
Factorial Designs
![Page 8: Experimental Designs for Optimizing InterventionsInbal (Billie) Nahum-Shani Experimental Designs for Optimizing Interventions mHealth Training Institute August 2016. 1 Key Definition](https://reader033.fdocuments.net/reader033/viewer/2022053106/606fe9d3c5c8310527717ef3/html5/thumbnails/8.jpg)
7
Factorial Designs Factorials: More than 1 factor; levels of each factor crossed
with levels of other factors.
►Should I include Text Messages?• Factor 1: Text (On/Off)
►Should I include Meal Replacement?• Factor 2: Meal (On/Off)
![Page 9: Experimental Designs for Optimizing InterventionsInbal (Billie) Nahum-Shani Experimental Designs for Optimizing Interventions mHealth Training Institute August 2016. 1 Key Definition](https://reader033.fdocuments.net/reader033/viewer/2022053106/606fe9d3c5c8310527717ef3/html5/thumbnails/9.jpg)
8
Factorial Designs Factorials: More than 1 factor; levels of each factor crossed
with levels of other factors.
►Should I include Text Messages?• Factor 1: Text (On/Off)
►Should I include Meal Replacement?• Factor 2: Meal (On/Off)
Experimental conditions 2X2 factorial N=400
Experiment Condition
Factor
Text Meal
1 (N=100) On On
2 (N=100) On Off
3 (N=100) Off On
4 (N=100) Off Off
![Page 10: Experimental Designs for Optimizing InterventionsInbal (Billie) Nahum-Shani Experimental Designs for Optimizing Interventions mHealth Training Institute August 2016. 1 Key Definition](https://reader033.fdocuments.net/reader033/viewer/2022053106/606fe9d3c5c8310527717ef3/html5/thumbnails/10.jpg)
9
Factorial Designs Factorials: More than 1 factor; levels of each factor crossed
with levels of other factors.
►Should I include Text Messages?• Factor 1: Text (On/Off)• Main effect: On (N=200) vs. Off (N=200)
►Should I include Meal Replacement?• Factor 2: Meal (On/Off)
Experimental conditions 2X2 factorial N=400
Experiment Condition
Factor
Text Meal
1 (N=100) On On
2 (N=100) On Off
3 (N=100) Off On
4 (N=100) Off Off
![Page 11: Experimental Designs for Optimizing InterventionsInbal (Billie) Nahum-Shani Experimental Designs for Optimizing Interventions mHealth Training Institute August 2016. 1 Key Definition](https://reader033.fdocuments.net/reader033/viewer/2022053106/606fe9d3c5c8310527717ef3/html5/thumbnails/11.jpg)
10
Factorial Designs Factorials: More than 1 factor; levels of each factor crossed
with levels of other factors.
Q1. Should I include Text Messages?• Factor 1: Text (On/Off)• Main effect: On (N=200) vs. Off (N=200)
Q2. Should I include Meal Replacement?• Factor 2: Meal (On/Off)• Main effect: On (N=200) vs. Off (N=200)
Experimental conditions 2X2 factorial N=400
Experiment Condition
Factor
Text Meal
1 (N=100) On On
2 (N=100) On Off
3 (N=100) Off On
4 (N=100) Off Off
![Page 12: Experimental Designs for Optimizing InterventionsInbal (Billie) Nahum-Shani Experimental Designs for Optimizing Interventions mHealth Training Institute August 2016. 1 Key Definition](https://reader033.fdocuments.net/reader033/viewer/2022053106/606fe9d3c5c8310527717ef3/html5/thumbnails/12.jpg)
11
Factorial Designs Factorials: More than 1 factor; levels of each factor crossed
with levels of other factors.
Q1. Should I include Text Messages?• Factor 1: Text (On/Off)
Q2. Should I include Meal Replacement?• Factor 2: Meal (On/Off)
Q3. Should I include Buddy Training?• Factor 3: Buddy (On/Off)
![Page 13: Experimental Designs for Optimizing InterventionsInbal (Billie) Nahum-Shani Experimental Designs for Optimizing Interventions mHealth Training Institute August 2016. 1 Key Definition](https://reader033.fdocuments.net/reader033/viewer/2022053106/606fe9d3c5c8310527717ef3/html5/thumbnails/13.jpg)
12
Factorial Designs Factorials: More than 1 factor; levels of each factor crossed
with levels of other factors.Experimental conditions 2X2X2
factorial N=400
Condition Factor
Text Meal Buddy
1 (N=50) On On On
2 (N=50) On On Off
3 (N=50) On Off On
4 (N=50) On Off Off
5 (N=50) Off On On
6 (N=50) Off On Off
7 (N=50) Off Off On
8 (N=50) Off Off Off
Q1. Should I include Text Messages?• Factor 1: Text (On/Off)
Q2. Should I include Meal Replacement?• Factor 2: Meal (On/Off)
Q3. Should I include Buddy Training?• Factor 3: Buddy (On/Off)
![Page 14: Experimental Designs for Optimizing InterventionsInbal (Billie) Nahum-Shani Experimental Designs for Optimizing Interventions mHealth Training Institute August 2016. 1 Key Definition](https://reader033.fdocuments.net/reader033/viewer/2022053106/606fe9d3c5c8310527717ef3/html5/thumbnails/14.jpg)
13
Factorial Designs Factorials: More than 1 factor; levels of each factor crossed
with levels of other factors.Experimental conditions 2X2X2
factorial N=400
Condition Factor
Text Meal Buddy
1 (N=50) On On On
2 (N=50) On On Off
3 (N=50) On Off On
4 (N=50) On Off Off
5 (N=50) Off On On
6 (N=50) Off On Off
7 (N=50) Off Off On
8 (N=50) Off Off Off
Q1. Should I include Text Messages?• Factor 1: Text (On/Off)• Main effect: On (N=200) vs. Off (N=200)
Q2. Should I include Meal Replacement?• Factor 2: Meal (On/Off)
Q3. Should I include Buddy Training?• Factor 3: Buddy (On/Off)
![Page 15: Experimental Designs for Optimizing InterventionsInbal (Billie) Nahum-Shani Experimental Designs for Optimizing Interventions mHealth Training Institute August 2016. 1 Key Definition](https://reader033.fdocuments.net/reader033/viewer/2022053106/606fe9d3c5c8310527717ef3/html5/thumbnails/15.jpg)
14
Factorial Designs Factorials: More than 1 factor; levels of each factor crossed
with levels of other factors.Experimental conditions 2X2X2
factorial N=400
Condition Factor
Text Meal Buddy
1 (N=50) On On On
2 (N=50) On On Off
3 (N=50) On Off On
4 (N=50) On Off Off
5 (N=50) Off On On
6 (N=50) Off On Off
7 (N=50) Off Off On
8 (N=50) Off Off Off
Q1. Should I include Text Messages?• Factor 1: Text (On/Off)
Q2. Should I include Meal Replacement?• Factor 2: Meal (On/Off)• Main effect: On (N=200) vs. Off (N=200)
Q3. Should I include Buddy Training?• Factor 3: Buddy (On/Off)
![Page 16: Experimental Designs for Optimizing InterventionsInbal (Billie) Nahum-Shani Experimental Designs for Optimizing Interventions mHealth Training Institute August 2016. 1 Key Definition](https://reader033.fdocuments.net/reader033/viewer/2022053106/606fe9d3c5c8310527717ef3/html5/thumbnails/16.jpg)
15
Factorial Designs Factorials: More than 1 factor; levels of each factor crossed
with levels of other factors.Experimental conditions 2X2X2
factorial N=400
Condition Factor
Text Meal Buddy
1 (N=50) On On On
2 (N=50) On On Off
3 (N=50) On Off On
4 (N=50) On Off Off
5 (N=50) Off On On
6 (N=50) Off On Off
7 (N=50) Off Off On
8 (N=50) Off Off Off
Q1. Should I include Text Messages?• Factor 1: Text (On/Off)
Q2. Should I include Meal Replacement?• Factor 2: Meal (On/Off)
Q3. Should I include Buddy Training?• Factor 3: Buddy (On/Off)• Main effect: On (N=200) vs. Off (N=200)
![Page 17: Experimental Designs for Optimizing InterventionsInbal (Billie) Nahum-Shani Experimental Designs for Optimizing Interventions mHealth Training Institute August 2016. 1 Key Definition](https://reader033.fdocuments.net/reader033/viewer/2022053106/606fe9d3c5c8310527717ef3/html5/thumbnails/17.jpg)
16
Factorial Designs Power for comparing package vs. control?
Experimental conditions 2X2X2 factorial N=400
Condition Factor
Text Meal Buddy
1 (N=50) On On On
2 (N=50) On On Off
3 (N=50) On Off On
4 (N=50) On Off Off
5 (N=50) Off On On
6 (N=50) Off On Off
7 (N=50) Off Off On
8 (N=50) Off Off Off
Experimental conditions 2X2 factorial N=400
Experiment Condition
Factor
Text Meal
1 (N=100) On On
2 (N=100) On Off
3 (N=100) Off On
4 (N=100) Off Off
![Page 18: Experimental Designs for Optimizing InterventionsInbal (Billie) Nahum-Shani Experimental Designs for Optimizing Interventions mHealth Training Institute August 2016. 1 Key Definition](https://reader033.fdocuments.net/reader033/viewer/2022053106/606fe9d3c5c8310527717ef3/html5/thumbnails/18.jpg)
17
SMART Designs
![Page 19: Experimental Designs for Optimizing InterventionsInbal (Billie) Nahum-Shani Experimental Designs for Optimizing Interventions mHealth Training Institute August 2016. 1 Key Definition](https://reader033.fdocuments.net/reader033/viewer/2022053106/606fe9d3c5c8310527717ef3/html5/thumbnails/19.jpg)
18
SMART Designs Adaptive Intervention:
– Intervention design that uses ongoing/dynamic information about the individual to decide which component to offer, when and how.
Hypothetical Example: (NIH/NIDDK R01DK108678; Spring & Nahum-Shani)
Text
Add Buddy
Step-down
Week 0 Week 2
Non-Responders
Responders
Stage 1 = {Text}, ThenIF response = {NO}
THEN stage 2 = {Add Buddy} ELSE IF response = {YES}
THEN stage 2 = {Step-Down}
![Page 20: Experimental Designs for Optimizing InterventionsInbal (Billie) Nahum-Shani Experimental Designs for Optimizing Interventions mHealth Training Institute August 2016. 1 Key Definition](https://reader033.fdocuments.net/reader033/viewer/2022053106/606fe9d3c5c8310527717ef3/html5/thumbnails/20.jpg)
19
SMART Designs Motivation in the context of technology-supported interventions:
– Boredom: Lack of interest in and difficulty concentrating on the task.
– Burden: The “workload” required from participants and the impact on their well-being.
– Cost: Some mHealth components are costly; resources are often limited
![Page 21: Experimental Designs for Optimizing InterventionsInbal (Billie) Nahum-Shani Experimental Designs for Optimizing Interventions mHealth Training Institute August 2016. 1 Key Definition](https://reader033.fdocuments.net/reader033/viewer/2022053106/606fe9d3c5c8310527717ef3/html5/thumbnails/21.jpg)
20
SMART Designs SMARTs can help us build empirically-based adaptive interventions:
– Randomized Trials – Multiple stages of randomization– Each stage corresponds to a critical question concerning the sequencing and
adaptation of intervention options over time
![Page 22: Experimental Designs for Optimizing InterventionsInbal (Billie) Nahum-Shani Experimental Designs for Optimizing Interventions mHealth Training Institute August 2016. 1 Key Definition](https://reader033.fdocuments.net/reader033/viewer/2022053106/606fe9d3c5c8310527717ef3/html5/thumbnails/22.jpg)
21
Hypothetical Example (NIH/NIDDK R01DK108678; Spring & Nahum-Shani)
– Remember this Adaptive Intervention?
SMART Designs
Text
Add Buddy
Step-down
Week 0 Week 2
Non-Responders
Responders
Stage 1 = {Text}, ThenIF response = {NO}
THEN stage 2 = {Add Buddy} ELSE IF response = {YES}
THEN stage 2 = {Step-Down}
![Page 23: Experimental Designs for Optimizing InterventionsInbal (Billie) Nahum-Shani Experimental Designs for Optimizing Interventions mHealth Training Institute August 2016. 1 Key Definition](https://reader033.fdocuments.net/reader033/viewer/2022053106/606fe9d3c5c8310527717ef3/html5/thumbnails/23.jpg)
22
SMART Designs Hypothetical Example (NIH/NIDDK R01DK108678; Spring & Nahum-Shani)
– Aim: Develop an adaptive technology-supported weight loss intervention
– Open scientific questionsQ1. Which component to offer first: Text or Phone? Q2. Which component to add for non-responders: Buddy or Phone?
![Page 24: Experimental Designs for Optimizing InterventionsInbal (Billie) Nahum-Shani Experimental Designs for Optimizing Interventions mHealth Training Institute August 2016. 1 Key Definition](https://reader033.fdocuments.net/reader033/viewer/2022053106/606fe9d3c5c8310527717ef3/html5/thumbnails/24.jpg)
23
Questions We Can Address with SMART
First-stage intervention component:– Is it better to start with Phone Coaching or Text Messages?– (SG1+SG2+SG3) vs. (SG4+SG5+SG6) – Phone Coaching vs. Text Messages
• Controlling for subsequent intervention component
R
Phone
Text
Response
Non-Response RBuddy (SG3)
Phone (SG2)
Response
Non-Response RBuddy (SG6)
Phone (SG5)
Step-Down (SG1)
Step-down (SG4)
n=200
n=200N=400
![Page 25: Experimental Designs for Optimizing InterventionsInbal (Billie) Nahum-Shani Experimental Designs for Optimizing Interventions mHealth Training Institute August 2016. 1 Key Definition](https://reader033.fdocuments.net/reader033/viewer/2022053106/606fe9d3c5c8310527717ef3/html5/thumbnails/25.jpg)
24
Questions We Can Address with SMART Second-stage intervention component:
– For non-responders: Is it better to add Phone or Buddy?– (SG2+SG5) vs. (SG3+SG6) – Phone Coaching vs. Buddy Training
R
Phone
Text
Response
Non-Response RBuddy (SG3)
Phone (SG2)
Response
Non-Response RBuddy (SG6)
Phone (SG5)
Step-Down (SG1)
Step-Down (SG4)
N=400 n=100
n=100
50%
50%
![Page 26: Experimental Designs for Optimizing InterventionsInbal (Billie) Nahum-Shani Experimental Designs for Optimizing Interventions mHealth Training Institute August 2016. 1 Key Definition](https://reader033.fdocuments.net/reader033/viewer/2022053106/606fe9d3c5c8310527717ef3/html5/thumbnails/26.jpg)
25
Questions We Can Address with SMART Embedded adaptive interventions
R
Phone
Text
Response
Non-Response RBuddy (SG3)
Phone (SG2)
Response
Non-Response RBuddy (SG6)
Phone (SG5)
Step-Down (SG1)
Step-Down (SG4)
Stage 1 = {Text}, ThenIF response = {NO}
THEN stage 2 = {Add Buddy} ELSE IF response = {YES}
THEN stage 2 = {Step-Down}
![Page 27: Experimental Designs for Optimizing InterventionsInbal (Billie) Nahum-Shani Experimental Designs for Optimizing Interventions mHealth Training Institute August 2016. 1 Key Definition](https://reader033.fdocuments.net/reader033/viewer/2022053106/606fe9d3c5c8310527717ef3/html5/thumbnails/27.jpg)
26
Questions We Can Address with SMART Embedded adaptive interventions
Stage 1 = {Text}, ThenIF response = {NO}
THEN stage 2 = {Add Phone} ELSE IF response = {YES}
THEN stage 2 = {Step-Down}
R
Phone
Text
Response
Non-Response RBuddy (SG3)
Phone (SG2)
Response
Non-Response RBuddy (SG6)
Phone (SG5)
Step-Down (SG1)
Step-Down (SG4)
![Page 28: Experimental Designs for Optimizing InterventionsInbal (Billie) Nahum-Shani Experimental Designs for Optimizing Interventions mHealth Training Institute August 2016. 1 Key Definition](https://reader033.fdocuments.net/reader033/viewer/2022053106/606fe9d3c5c8310527717ef3/html5/thumbnails/28.jpg)
27
Questions We Can Address with SMART Embedded adaptive interventions
R
Phone
Text
Response
Non-Response RBuddy (SG3)
Phone (SG2)
Response
Non-Response RBuddy (SG6)
Phone (SG5)
Step-Down (SG1)
Step-Down (SG4)
Stage 1 = {Phone}, ThenIF response = {NO}
THEN stage 2 = {Add Phone} ELSE IF response = {YES}
THEN stage 2= {Step-Down}
![Page 29: Experimental Designs for Optimizing InterventionsInbal (Billie) Nahum-Shani Experimental Designs for Optimizing Interventions mHealth Training Institute August 2016. 1 Key Definition](https://reader033.fdocuments.net/reader033/viewer/2022053106/606fe9d3c5c8310527717ef3/html5/thumbnails/29.jpg)
28
Questions We Can Address with SMART Embedded adaptive interventions
R
Phone
Text
Response
Non-Response RBuddy (SG3)
Phone (SG2)
Response
Non-Response RBuddy (SG6)
Phone (SG5)
Step-Down (SG1)
Step-Down (SG4)
Stage 1 = {Phone}Then, IF response = {NO}
THEN stage 2 = {Add Buddy}ELSE IF response = {YES}
THEN stage 2= {Step-Down}
![Page 30: Experimental Designs for Optimizing InterventionsInbal (Billie) Nahum-Shani Experimental Designs for Optimizing Interventions mHealth Training Institute August 2016. 1 Key Definition](https://reader033.fdocuments.net/reader033/viewer/2022053106/606fe9d3c5c8310527717ef3/html5/thumbnails/30.jpg)
29
Questions We Can Address with SMART Embedded adaptive interventions
Stage 1 = {Text}, ThenIF response = {NO}
THEN stage 2 = {Add Buddy} ELSE IF response = {YES}
THEN stage 2 = {Step-Down}
VS.
Stage 1 = {Phone}, ThenIF response = {NO}
THEN stage 2 = {Add Phone} ELSE IF response = {YES}
THEN stage 2= {Step-Down}
R
Phone
Text
Response
Non-Response RBuddy (SG3)
Phone (SG2)
Response
Non-Response RBuddy (SG6)
Phone (SG5)
Step-Down (SG1)
Step-Down (SG4)
![Page 31: Experimental Designs for Optimizing InterventionsInbal (Billie) Nahum-Shani Experimental Designs for Optimizing Interventions mHealth Training Institute August 2016. 1 Key Definition](https://reader033.fdocuments.net/reader033/viewer/2022053106/606fe9d3c5c8310527717ef3/html5/thumbnails/31.jpg)
30
Summary Factorial Designs:
Efficacy of Individual components
– Which components are effective?– Which level is more appropriate?– Which components work well together?
SMART Designs:
Sequencing and adaptation of components
– Which component to offer first?– Which to offer subsequently?– How should I tailor components over time?
![Page 32: Experimental Designs for Optimizing InterventionsInbal (Billie) Nahum-Shani Experimental Designs for Optimizing Interventions mHealth Training Institute August 2016. 1 Key Definition](https://reader033.fdocuments.net/reader033/viewer/2022053106/606fe9d3c5c8310527717ef3/html5/thumbnails/32.jpg)
31
Experts + Resources Collaborators:
– U of Michigan: Statistical Reinforcement Learning Lab • Susan Murphy: http://dept.stat.lsa.umich.edu/~samurphy/• Danny Almirall: http://www-personal.umich.edu/~dalmiral/
– Penn State: Methodology Center• Linda Collins: http://methodology.psu.edu/people/lcollins• John Dziak: http://methodology.psu.edu/people/jdziak
Resources:
– SMART:• Projects using SMARTs: https://methodology.psu.edu/ra/adap-inter• Lei, H., Nahum-Shani, I., Lynch, K., Oslin, D., & Murphy, S. A. (2012). A "SMART" design for building
individualized treatment sequences. Annual Review of Clinical Psychology, 8, 14.1 - 14.28
– Factorials: • Q&A: https://methodology.psu.edu/ra/most/fefaq• Collins, L. M., Dziak, J. J., & Li, R. (2009). Design of experiments with multiple independent variables: A
resource management perspective on complete and reduced factorial designs. Psychological Methods, 14, 202-224.