Responsive Design for Household Surveys: Illustration of Management Interventions Based on Survey...
-
Upload
donald-allison -
Category
Documents
-
view
216 -
download
0
Transcript of Responsive Design for Household Surveys: Illustration of Management Interventions Based on Survey...
![Page 1: Responsive Design for Household Surveys: Illustration of Management Interventions Based on Survey Paradata Robert M. Groves, Emilia Peytcheva, Nicole Kirgis,](https://reader036.fdocuments.net/reader036/viewer/2022062717/56649e445503460f94b37cb5/html5/thumbnails/1.jpg)
Responsive Design for Household Surveys:
Illustration of Management Interventions Based on Survey Paradata
Robert M. Groves, Emilia Peytcheva, Nicole Kirgis, James Wagner, William Axinn, University of Michigan,
USAWilliam Mosher, US National Center for Health
Statistics
Research partially supported by contract with the US National Center for Health Statistics, Contract No. 200-2000-07001
![Page 2: Responsive Design for Household Surveys: Illustration of Management Interventions Based on Survey Paradata Robert M. Groves, Emilia Peytcheva, Nicole Kirgis,](https://reader036.fdocuments.net/reader036/viewer/2022062717/56649e445503460f94b37cb5/html5/thumbnails/2.jpg)
Definition: Responsive Design
1. Preidentify a set of alternative features potentially affecting costs and errors of statistics
2. Identify a set of indicators of the cost and error properties of those feature
3. Monitor indicators in initial stages of data collection
4. Alter the active features of the survey based on cost/error tradeoff decision rules
5. Combine data from separate phases into a single estimator
Survey designs that:
![Page 3: Responsive Design for Household Surveys: Illustration of Management Interventions Based on Survey Paradata Robert M. Groves, Emilia Peytcheva, Nicole Kirgis,](https://reader036.fdocuments.net/reader036/viewer/2022062717/56649e445503460f94b37cb5/html5/thumbnails/3.jpg)
The Why’s of Responsive Designs
• One-off surveys are mounted with large uncertainties (e.g., eligibility of frame elements, effort required to contact, cooperation rate, length of interview)
• Most survey budgets are relatively fixed at start of project
• Some survey errors are functions of effort during production
Hence, quality is out of control of researcher unless designs are permitted to change based on production experience
![Page 4: Responsive Design for Household Surveys: Illustration of Management Interventions Based on Survey Paradata Robert M. Groves, Emilia Peytcheva, Nicole Kirgis,](https://reader036.fdocuments.net/reader036/viewer/2022062717/56649e445503460f94b37cb5/html5/thumbnails/4.jpg)
Effort
Daily Main and Screener HPI (total hours/ total I'ws) and 7-day Moving Average Main and Screener HPI by Day of Survey
0
5
10
15
20
25
30
35
1 8 15 22 29 36 43 50 57 64 71 78Q1 7-day Average Main HPI Q3 Daily Main HPI Q2 7-day Average Main HPIQ1 7-day Average Scrn HPI Q3 Daily Scrn HPI Q2 7-day Average Scrn HPIQ3 7-day Average Screener Q3 7-day Main HPI
Phase 1 Phase 2
The NSFG Dashboard
hours
% production
Active Sample
nonworked
interviews
Productivity
calls/day
% peak callsscrn’r/main calls
mean calls
8+ calls
locked bldgs
resistant
hard appt.
occupied
eligible
propensity
calls/interview
hours/interview
% with kids
response rate
Daily Main and Screener HPI (total hours/ total I'ws) and 7-day Moving Average Main and Screener HPI by Day of Survey
0
5
10
15
20
25
30
35
1 8 15 22 29 36 43 50 57 64 71 78Q1 7-day Average Main HPI Q3 Daily Main HPI Q2 7-day Average Main HPIQ1 7-day Average Scrn HPI Q3 Daily Scrn HPI Q2 7-day Average Scrn HPIQ3 7-day Average Screener Q3 7-day Main HPI
Phase 1 Phase 2
I’rs working
Daily Main and Screener HPI (total hours/ total I'ws) and 7-day Moving Average Main and Screener HPI by Day of Survey
0
5
10
15
20
25
30
35
1 8 15 22 29 36 43 50 57 64 71 78Q1 7-day Average Main HPI Q3 Daily Main HPI Q2 7-day Average Main HPIQ1 7-day Average Scrn HPI Q3 Daily Scrn HPI Q2 7-day Average Scrn HPIQ3 7-day Average Screener Q3 7-day Main HPI
Phase 1 Phase 2
Daily Main and Screener HPI (total hours/ total I'ws) and 7-day Moving Average Main and Screener HPI by Day of Survey
0
5
10
15
20
25
30
35
1 8 15 22 29 36 43 50 57 64 71 78Q1 7-day Average Main HPI Q3 Daily Main HPI Q2 7-day Average Main HPIQ1 7-day Average Scrn HPI Q3 Daily Scrn HPI Q2 7-day Average Scrn HPIQ3 7-day Average Screener Q3 7-day Main HPI
Phase 1 Phase 2
Daily Main and Screener HPI (total hours/ total I'ws) and 7-day Moving Average Main and Screener HPI by Day of Survey
0
5
10
15
20
25
30
35
1 8 15 22 29 36 43 50 57 64 71 78Q1 7-day Average Main HPI Q3 Daily Main HPI Q2 7-day Average Main HPIQ1 7-day Average Scrn HPI Q3 Daily Scrn HPI Q2 7-day Average Scrn HPIQ3 7-day Average Screener Q3 7-day Main HPI
Phase 1 Phase 2
calls/hourDaily Main and Screener HPI (total hours/ total I'ws) and 7-day Moving Average Main and
Screener HPI by Day of Survey
0
5
10
15
20
25
30
35
1 8 15 22 29 36 43 50 57 64 71 78Q1 7-day Average Main HPI Q3 Daily Main HPI Q2 7-day Average Main HPIQ1 7-day Average Scrn HPI Q3 Daily Scrn HPI Q2 7-day Average Scrn HPIQ3 7-day Average Screener Q3 7-day Main HPI
Phase 1 Phase 2
Daily Main and Screener HPI (total hours/ total I'ws) and 7-day Moving Average Main and Screener HPI by Day of Survey
0
5
10
15
20
25
30
35
1 8 15 22 29 36 43 50 57 64 71 78Q1 7-day Average Main HPI Q3 Daily Main HPI Q2 7-day Average Main HPIQ1 7-day Average Scrn HPI Q3 Daily Scrn HPI Q2 7-day Average Scrn HPIQ3 7-day Average Screener Q3 7-day Main HPI
Phase 1 Phase 2
Daily Main and Screener HPI (total hours/ total I'ws) and 7-day Moving Average Main and Screener HPI by Day of Survey
0
5
10
15
20
25
30
35
1 8 15 22 29 36 43 50 57 64 71 78Q1 7-day Average Main HPI Q3 Daily Main HPI Q2 7-day Average Main HPIQ1 7-day Average Scrn HPI Q3 Daily Scrn HPI Q2 7-day Average Scrn HPIQ3 7-day Average Screener Q3 7-day Main HPI
Phase 1 Phase 2
Daily Main and Screener HPI (total hours/ total I'ws) and 7-day Moving Average Main and Screener HPI by Day of Survey
0
5
10
15
20
25
30
35
1 8 15 22 29 36 43 50 57 64 71 78Q1 7-day Average Main HPI Q3 Daily Main HPI Q2 7-day Average Main HPIQ1 7-day Average Scrn HPI Q3 Daily Scrn HPI Q2 7-day Average Scrn HPIQ3 7-day Average Screener Q3 7-day Main HPI
Phase 1 Phase 2
Daily Main and Screener HPI (total hours/ total I'ws) and 7-day Moving Average Main and Screener HPI by Day of Survey
0
5
10
15
20
25
30
35
1 8 15 22 29 36 43 50 57 64 71 78Q1 7-day Average Main HPI Q3 Daily Main HPI Q2 7-day Average Main HPIQ1 7-day Average Scrn HPI Q3 Daily Scrn HPI Q2 7-day Average Scrn HPIQ3 7-day Average Screener Q3 7-day Main HPI
Phase 1 Phase 2
Daily Main and Screener HPI (total hours/ total I'ws) and 7-day Moving Average Main and Screener HPI by Day of Survey
0
5
10
15
20
25
30
35
1 8 15 22 29 36 43 50 57 64 71 78Q1 7-day Average Main HPI Q3 Daily Main HPI Q2 7-day Average Main HPIQ1 7-day Average Scrn HPI Q3 Daily Scrn HPI Q2 7-day Average Scrn HPIQ3 7-day Average Screener Q3 7-day Main HPI
Phase 1 Phase 2
Daily Main and Screener HPI (total hours/ total I'ws) and 7-day Moving Average Main and Screener HPI by Day of Survey
0
5
10
15
20
25
30
35
1 8 15 22 29 36 43 50 57 64 71 78Q1 7-day Average Main HPI Q3 Daily Main HPI Q2 7-day Average Main HPIQ1 7-day Average Scrn HPI Q3 Daily Scrn HPI Q2 7-day Average Scrn HPIQ3 7-day Average Screener Q3 7-day Main HPI
Phase 1 Phase 2
Daily Main and Screener HPI (total hours/ total I'ws) and 7-day Moving Average Main and Screener HPI by Day of Survey
0
5
10
15
20
25
30
35
1 8 15 22 29 36 43 50 57 64 71 78Q1 7-day Average Main HPI Q3 Daily Main HPI Q2 7-day Average Main HPIQ1 7-day Average Scrn HPI Q3 Daily Scrn HPI Q2 7-day Average Scrn HPIQ3 7-day Average Screener Q3 7-day Main HPI
Phase 1 Phase 2
Daily Main and Screener HPI (total hours/ total I'ws) and 7-day Moving Average Main and Screener HPI by Day of Survey
0
5
10
15
20
25
30
35
1 8 15 22 29 36 43 50 57 64 71 78Q1 7-day Average Main HPI Q3 Daily Main HPI Q2 7-day Average Main HPIQ1 7-day Average Scrn HPI Q3 Daily Scrn HPI Q2 7-day Average Scrn HPIQ3 7-day Average Screener Q3 7-day Main HPI
Phase 1 Phase 2
Daily Main and Screener HPI (total hours/ total I'ws) and 7-day Moving Average Main and Screener HPI by Day of Survey
0
5
10
15
20
25
30
35
1 8 15 22 29 36 43 50 57 64 71 78Q1 7-day Average Main HPI Q3 Daily Main HPI Q2 7-day Average Main HPIQ1 7-day Average Scrn HPI Q3 Daily Scrn HPI Q2 7-day Average Scrn HPIQ3 7-day Average Screener Q3 7-day Main HPI
Phase 1 Phase 2
Daily Main and Screener HPI (total hours/ total I'ws) and 7-day Moving Average Main and Screener HPI by Day of Survey
0
5
10
15
20
25
30
35
1 8 15 22 29 36 43 50 57 64 71 78Q1 7-day Average Main HPI Q3 Daily Main HPI Q2 7-day Average Main HPIQ1 7-day Average Scrn HPI Q3 Daily Scrn HPI Q2 7-day Average Scrn HPIQ3 7-day Average Screener Q3 7-day Main HPI
Phase 1 Phase 2
Daily Main and Screener HPI (total hours/ total I'ws) and 7-day Moving Average Main and Screener HPI by Day of Survey
0
5
10
15
20
25
30
35
1 8 15 22 29 36 43 50 57 64 71 78Q1 7-day Average Main HPI Q3 Daily Main HPI Q2 7-day Average Main HPIQ1 7-day Average Scrn HPI Q3 Daily Scrn HPI Q2 7-day Average Scrn HPIQ3 7-day Average Screener Q3 7-day Main HPI
Phase 1 Phase 2
noncontacts
Daily Main and Screener HPI (total hours/ total I'ws) and 7-day Moving Average Main and Screener HPI by Day of Survey
0
5
10
15
20
25
30
35
1 8 15 22 29 36 43 50 57 64 71 78Q1 7-day Average Main HPI Q3 Daily Main HPI Q2 7-day Average Main HPIQ1 7-day Average Scrn HPI Q3 Daily Scrn HPI Q2 7-day Average Scrn HPIQ3 7-day Average Screener Q3 7-day Main HPI
Phase 1 Phase 2
Daily Main and Screener HPI (total hours/ total I'ws) and 7-day Moving Average Main and Screener HPI by Day of Survey
0
5
10
15
20
25
30
35
1 8 15 22 29 36 43 50 57 64 71 78Q1 7-day Average Main HPI Q3 Daily Main HPI Q2 7-day Average Main HPIQ1 7-day Average Scrn HPI Q3 Daily Scrn HPI Q2 7-day Average Scrn HPIQ3 7-day Average Screener Q3 7-day Main HPI
Phase 1 Phase 2
Daily Main and Screener HPI (total hours/ total I'ws) and 7-day Moving Average Main and Screener HPI by Day of Survey
0
5
10
15
20
25
30
35
1 8 15 22 29 36 43 50 57 64 71 78Q1 7-day Average Main HPI Q3 Daily Main HPI Q2 7-day Average Main HPIQ1 7-day Average Scrn HPI Q3 Daily Scrn HPI Q2 7-day Average Scrn HPIQ3 7-day Average Screener Q3 7-day Main HPI
Phase 1 Phase 2
cum. interviews
Daily Main and Screener HPI (total hours/ total I'ws) and 7-day Moving Average Main and Screener HPI by Day of Survey
0
5
10
15
20
25
30
35
1 8 15 22 29 36 43 50 57 64 71 78Q1 7-day Average Main HPI Q3 Daily Main HPI Q2 7-day Average Main HPIQ1 7-day Average Scrn HPI Q3 Daily Scrn HPI Q2 7-day Average Scrn HPIQ3 7-day Average Screener Q3 7-day Main HPI
Phase 1 Phase 2
Data Set BalanceDaily Main and Screener HPI (total hours/ total I'ws) and 7-day Moving Average Main and
Screener HPI by Day of Survey
0
5
10
15
20
25
30
35
1 8 15 22 29 36 43 50 57 64 71 78Q1 7-day Average Main HPI Q3 Daily Main HPI Q2 7-day Average Main HPIQ1 7-day Average Scrn HPI Q3 Daily Scrn HPI Q2 7-day Average Scrn HPIQ3 7-day Average Screener Q3 7-day Main HPI
Phase 1 Phase 2
Daily Main and Screener HPI (total hours/ total I'ws) and 7-day Moving Average Main and Screener HPI by Day of Survey
0
5
10
15
20
25
30
35
1 8 15 22 29 36 43 50 57 64 71 78Q1 7-day Average Main HPI Q3 Daily Main HPI Q2 7-day Average Main HPIQ1 7-day Average Scrn HPI Q3 Daily Scrn HPI Q2 7-day Average Scrn HPIQ3 7-day Average Screener Q3 7-day Main HPI
Phase 1 Phase 2
% sexually activeDaily Main and Screener HPI (total hours/ total I'ws) and 7-day Moving Average Main and
Screener HPI by Day of Survey
0
5
10
15
20
25
30
35
1 8 15 22 29 36 43 50 57 64 71 78Q1 7-day Average Main HPI Q3 Daily Main HPI Q2 7-day Average Main HPIQ1 7-day Average Scrn HPI Q3 Daily Scrn HPI Q2 7-day Average Scrn HPIQ3 7-day Average Screener Q3 7-day Main HPI
Phase 1 Phase 2
Daily Main and Screener HPI (total hours/ total I'ws) and 7-day Moving Average Main and Screener HPI by Day of Survey
0
5
10
15
20
25
30
35
1 8 15 22 29 36 43 50 57 64 71 78Q1 7-day Average Main HPI Q3 Daily Main HPI Q2 7-day Average Main HPIQ1 7-day Average Scrn HPI Q3 Daily Scrn HPI Q2 7-day Average Scrn HPIQ3 7-day Average Screener Q3 7-day Main HPI
Phase 1 Phase 2
group ratesDaily Main and Screener HPI (total hours/ total I'ws) and 7-day Moving Average Main and
Screener HPI by Day of Survey
0
5
10
15
20
25
30
35
1 8 15 22 29 36 43 50 57 64 71 78Q1 7-day Average Main HPI Q3 Daily Main HPI Q2 7-day Average Main HPIQ1 7-day Average Scrn HPI Q3 Daily Scrn HPI Q2 7-day Average Scrn HPIQ3 7-day Average Screener Q3 7-day Main HPI
Phase 1 Phase 2
CV group rates
X
![Page 5: Responsive Design for Household Surveys: Illustration of Management Interventions Based on Survey Paradata Robert M. Groves, Emilia Peytcheva, Nicole Kirgis,](https://reader036.fdocuments.net/reader036/viewer/2022062717/56649e445503460f94b37cb5/html5/thumbnails/5.jpg)
Evaluation of Two Interventions Based on Paradata
• Survey setting: face-to-face survey screening (3-5 min.) to locate 60% of households with 15-44 year old; one eligible sampled (60-80 min.)
• Interventions– Increasing relative effort on screening interviews
versus main interviews with selected respondent– Increasing relative effort on a small subset of cases
with high selection weights and high propensities to respond
![Page 6: Responsive Design for Household Surveys: Illustration of Management Interventions Based on Survey Paradata Robert M. Groves, Emilia Peytcheva, Nicole Kirgis,](https://reader036.fdocuments.net/reader036/viewer/2022062717/56649e445503460f94b37cb5/html5/thumbnails/6.jpg)
Ratio of Screener Calls to Main Interview Calls by Day by Quarter
0
1
2
3
4
5
6
1 8 15 22 29 36 43 50 57 64
Q2 7-day Average Scrn/Main calls Q1 7 day Scrn/Main calls Q3 7-day Scrn/Main Calls
Q4 7-day Scrn/Main Calls Year 1 Summary
![Page 7: Responsive Design for Household Surveys: Illustration of Management Interventions Based on Survey Paradata Robert M. Groves, Emilia Peytcheva, Nicole Kirgis,](https://reader036.fdocuments.net/reader036/viewer/2022062717/56649e445503460f94b37cb5/html5/thumbnails/7.jpg)
Autoregressive Time Series Coefficients for Model Predicting Daily Number of Screener Calls
(p-values for coefficients)
Q1 Q2 Q3 Q4
Intercept 279.9(<.0001)
385.0(<.0001)
345.3(<.0001)
294.8(<.0001)
Day -0.73(.48)
-3.2(0.28)
-2.1(.21)
-2.0(.17)
Screener Week
28.3(.29)
97.5(.009)
24.6(.56)
39.0(.29)
![Page 8: Responsive Design for Household Surveys: Illustration of Management Interventions Based on Survey Paradata Robert M. Groves, Emilia Peytcheva, Nicole Kirgis,](https://reader036.fdocuments.net/reader036/viewer/2022062717/56649e445503460f94b37cb5/html5/thumbnails/8.jpg)
Autoregressive Time Series Coefficients for Model Predicting Daily Number of Screener Interviews
(p-values for coefficients)
Q1 Q2 Q3 Q4
Intercept 63.6(<.0001)
94.5(<.0001)
64.6(<.0001)
63.4(<.0001)
Day -0.51(.032)
-1.23(0.001)
-0.54(.16)
-0.69(.09)
Screener Week
6.1(.30)
18.5(.01)
12.3(.20)
6.1(.54)
![Page 9: Responsive Design for Household Surveys: Illustration of Management Interventions Based on Survey Paradata Robert M. Groves, Emilia Peytcheva, Nicole Kirgis,](https://reader036.fdocuments.net/reader036/viewer/2022062717/56649e445503460f94b37cb5/html5/thumbnails/9.jpg)
Second Intervention: Increased Emphasis on Subset of Active Cases
• In the last weeks of Phase 1, a subsample of cases with high propensities and high selection weights are identified
• These cases are chosen to improve balance of respondent pool
• Interviewers are asked to give greater emphasis to these cases
![Page 10: Responsive Design for Household Surveys: Illustration of Management Interventions Based on Survey Paradata Robert M. Groves, Emilia Peytcheva, Nicole Kirgis,](https://reader036.fdocuments.net/reader036/viewer/2022062717/56649e445503460f94b37cb5/html5/thumbnails/10.jpg)
Mean Expected Probability to be Interviewed on Next Call, Screeners (Red) and Main (Green)
by Day of Data Collection by Quarter
0
0.05
0.1
0.15
0.2
0.25
0.3
0.35
0.4
1 8 15 22 29 36 43 50 57 64
Q1 Screener Mean Propensity Q1 Main Mean Propensity Q2 Screener Mean Propensity
Q2 Main Mean Propensity Q3 Screener Mean Propensity Q3 Main Mean Propensity
Q4 Screener Mean Propensity Q4 Main Mean Propensity Q2-Q4 Average Screener
Q2-Q4 Average Main
![Page 11: Responsive Design for Household Surveys: Illustration of Management Interventions Based on Survey Paradata Robert M. Groves, Emilia Peytcheva, Nicole Kirgis,](https://reader036.fdocuments.net/reader036/viewer/2022062717/56649e445503460f94b37cb5/html5/thumbnails/11.jpg)
Analytic Approach
• Not all interviewers active workloads contain both “intervention” cases and “non-intervention” cases
• We limit the analysis to those who have both types of cases
• We examine two indicators of success:– Mean number of calls (imperfect)– Response rate in intervention period
![Page 12: Responsive Design for Household Surveys: Illustration of Management Interventions Based on Survey Paradata Robert M. Groves, Emilia Peytcheva, Nicole Kirgis,](https://reader036.fdocuments.net/reader036/viewer/2022062717/56649e445503460f94b37cb5/html5/thumbnails/12.jpg)
Comparison of Mean Screener Calls During Intervention Period for Intervention and
Nonintervention Cases
Little evidence of increased calling on intervention cases
![Page 13: Responsive Design for Household Surveys: Illustration of Management Interventions Based on Survey Paradata Robert M. Groves, Emilia Peytcheva, Nicole Kirgis,](https://reader036.fdocuments.net/reader036/viewer/2022062717/56649e445503460f94b37cb5/html5/thumbnails/13.jpg)
Comparison of Screener Response Rate During Intervention Period for Intervention
and Nonintervention Cases
Little evidence of increased response rate on intervention cases
![Page 14: Responsive Design for Household Surveys: Illustration of Management Interventions Based on Survey Paradata Robert M. Groves, Emilia Peytcheva, Nicole Kirgis,](https://reader036.fdocuments.net/reader036/viewer/2022062717/56649e445503460f94b37cb5/html5/thumbnails/14.jpg)
Comparison of Mean Main Calls During Intervention Period for Intervention and
Nonintervention Cases
Mixed evidence on higher calls on intervention cases
![Page 15: Responsive Design for Household Surveys: Illustration of Management Interventions Based on Survey Paradata Robert M. Groves, Emilia Peytcheva, Nicole Kirgis,](https://reader036.fdocuments.net/reader036/viewer/2022062717/56649e445503460f94b37cb5/html5/thumbnails/15.jpg)
Comparison of Main Response Rate During Intervention Period for Intervention and
Nonintervention Cases
General tendency to higher response rates for interventioncases
![Page 16: Responsive Design for Household Surveys: Illustration of Management Interventions Based on Survey Paradata Robert M. Groves, Emilia Peytcheva, Nicole Kirgis,](https://reader036.fdocuments.net/reader036/viewer/2022062717/56649e445503460f94b37cb5/html5/thumbnails/16.jpg)
Logistic Regression For Likelihood of Main Interview
Quarter 2 Quarter 3 Quarter 4 Quarter 5Intercept -2.13*** -1.97*** -0.81*** -0.84**Intervention -0.07 0.72* 0.13 0.50*Weight 0.00 0.00 0.00 0.00Propensity 5.46*** 2.98** 2.02*** 0.65
![Page 17: Responsive Design for Household Surveys: Illustration of Management Interventions Based on Survey Paradata Robert M. Groves, Emilia Peytcheva, Nicole Kirgis,](https://reader036.fdocuments.net/reader036/viewer/2022062717/56649e445503460f94b37cb5/html5/thumbnails/17.jpg)
Conclusions
• Intervention 1: Management direction to focus on screeners vs. main increases calls, sometimes dramatically; screener interviews follow
• Intervention 2: Effectiveness at focusing on individual cases greater for main interviews than screener interviews
![Page 18: Responsive Design for Household Surveys: Illustration of Management Interventions Based on Survey Paradata Robert M. Groves, Emilia Peytcheva, Nicole Kirgis,](https://reader036.fdocuments.net/reader036/viewer/2022062717/56649e445503460f94b37cb5/html5/thumbnails/18.jpg)
Next Steps on Responsive Design with Paradata
• Responsive design requires effective central management direction of interviewer behavior
• We’re still learning how to communicate these directives consistently well