08/08/2015 Statistics Canada Statistique Canada Paradata Collection Research for Social Surveys at...
-
Upload
vincent-ward -
Category
Documents
-
view
229 -
download
0
Transcript of 08/08/2015 Statistics Canada Statistique Canada Paradata Collection Research for Social Surveys at...
23-04-19Statistics Canada • Statistique Canada
Paradata Collection Research for Social Surveys at Statistics Canada
François LaflammeInternational Total Survey Error Workshop (ITSEW)
Quebec, June 2011
23-04-19Statistics Canada • Statistique Canada2
OutlineData collection organization Data collection challengesParadata
Sources Database
Paradata ResearchObjectives ScopePast research Current and future plans
Data Collection Organization
3 regions and 8 locations across CanadaHO collects quarterly and annual business survey dataAll other business moved in the regions
Interviewers (~ 1,500) collect the following dataFace-to-face: CAPI/PAPI (~ monthly 100,000 attempts)
● Concurrent surveysCATI call centres (5) (~ monthly 900,000 calls)
● Household, agriculture and business surveys● Concurrent surveys
Unionization and operational constraints
3 Statistics Canada • Statistique Canada 23-04-19
4
Data Collection ChallengesHandling sophistication and increase of data
requirementsMaintaining acceptable response ratesEnsuring highest quality of data collectedOptimizing capacity
Balancing work within and between Regional OfficesRetention of workforce
Reducing / maintaining collection costsDeveloping and deploying surveys consistently, cost-
effectively and timelyKeeping abreast of evolving survey collection
methodologies and technologies (e.g. multi-mode surveys)
Taking into account operational constraints
Statistics Canada • Statistique Canada 23-04-19
23-04-19Statistics Canada • Statistique Canada5
Paradata SourcesParadata is Data Collection Process InformationParadata sources
Call and contact informationAudit trail (interview key strokes)
Interviewer administrative and payroll informationInterviewer notes and observations - Not used extensively
Can be enhanced withSample design and sample unit informationCapacity and planning assumptions
● Budget and target figures
Paradata from previous cycle or supplement surveys
23-04-19Statistics Canada • Statistique Canada6
Paradata DatabaseParadata Database includes:
Call/attempts information for both ● Computer-Assisted Telephone Interview (CATI) surveys
● Computer-Assisted Personal Interview (CAPI) surveysInterviewer payroll information
Processed and standardized informationRaw files always available
Historical information since 2003Updated on daily basisPrior to 2006, used for reporting purposes - not for researchAudit trail kept separately
7 23-04-19Statistics Canada • Statistique Canada7
Paradata Research
Paradata can be used for : Operational research (including survey management)
●Essentially before and during data collectionMethodological research
●Historically, the focus is after data collection (e.g. non-response and measurement errors)
Often ‘grey’ zone between the two types of researchNeed to make the link between operational and
methodological research
8 23-04-19Statistics Canada • Statistique Canada8
Paradata Research Objectives
Better understand data collection processIdentify potential operational efficienciesEvaluate new data collection initiativesProvide timely feedback and information
Data collection survey management (Active Management)
Maintain and improve data qualityImprove the way surveys are conducted and
managed
23-04-19Statistics Canada • Statistique Canada9
Paradata Research Scope
Initial focus onOn CATI social surveys
● RDD, cross-sectional, longitudinal surveysCall and contact information
Extended toCATI agriculture surveysCAPI surveysPayroll informationAudit trail
And more recently toBusiness surveys
23-04-19Statistics Canada • Statistique Canada10
Past ResearchInitial analysis
Effort spent: calls and system timeReaching respondents: contact rate, sequence of calls, best
time to call, contact versus interview, etc.
Active managementCustomized reportsDashboard of key survey performance indicators
Impact of cap on callsOn response rates, survey estimates and costs
Production and cost analysisRelationship between production and costProductivity indicators and survey cost analysis
23-04-19Statistics Canada • Statistique Canada11
Past Research 2..3Pace of interview (PoINT)CAPI surveys - Initial investigations
Basic analysis: attempts, time spent, contact ratesParadata quality and consistencyProductivity and cost relationshipInteraction between CAPI surveys
Responsive Collection Design for CATI surveysActive managementIdentify a series of new indicators to assess data collection
quality and performance (e.g. representativity, productivity and cost, responding potential of in-progress case measure)
Implementation - two pilots surveysAnalysis
23-04-19Statistics Canada • Statistique Canada12
Past Research 3..3Many ad hoc research projects
Interviewer productivity by level of experienceInteraction between concurrent surveysSystem time versus non-system time, etc.
Research increased knowledge about data collection process and practicesDemonstrate potential benefits - Based on facts (empirical data)Investigate, test and implement new collection strategies and
tools - Think outside the box
Make the balance between theory and practice● Focus on operationally viable projects
Communicate and share information ● Documentation, papers, presentations, seminars, etc.
23-04-19Statistics Canada • Statistique Canada1313
Distribution of Calls and Timeby Collection Phase More calls and system time spent after a first contact
for both respondents and non-respondents
SLID 2006 - Distribution of Calls and Time Spent by Data Collection Phase
18%
35%
8% 7%
22%
10%10%
29%
37%
4%
13%8%
0%
10%
20%
30%
40%
50%
Before f irstcontact
Betw een firstcontact &interview
Interview Before f irstcontact
After the f irstcontact
No contact
Response Non-Response
% Calls
% System Time Spent
23-04-19Statistics Canada • Statistique Canada1414
Relationship between Production and Cost Throughout Survey Cycle
Strong relationship Most distributions have the same shapeSystem time is a good predictor for payroll hours Ratios of cost to production can be used to derive productivity indicators
Distribution of System Time and Payroll Hours by Collection Day - Survey of Labour and Income Dynamics (SLID 2007)
0
200
400
600
800
1,000
1,200
1 6 11 16 21 26 31 36 41 46 51 56 61 66 71 76 81 86 91 96 101Data Collection Days
Hours
Complete Interview System TimeTotal System TimeDirect Collection Payroll HoursTotal Payroll Hours
23-04-19Statistics Canada • Statistique Canada15
Survey Productivity Indicators Based on time Completed Interview System Time / Total System Time Ratios
Productivity ratios decrease during collection period for CATI Longitudinal CATI survey (SLID) shows larger decreases Productivity for CAPI survey is higher and more stable This ratio is affected by interview length and response rate
Distribution of Complete Interview System Time and Total Sytem Time Ratios For SLID (2008), CCHS CATI and CAPI (January-February 2008)
0%
20%
40%
60%
80%
100%
1 6 11 16 21 26 31 36 41 46 51 56 61 66 71Data Collection Days
CCHS - CAPI
CCHS - CATI
SLID
23-04-19Statistics Canada • Statistique Canada1616 23-04-19Statistics Canada • Statistique Canada16
Current and Future Research Plans
Focus on “Strategies to improve the way data collection is conducted and managed”.
Hence the research need to
Be sound and operationally viable Lead to more cost-effective collection and sample
design strategies Lead to data quality improvements
23-04-19Statistics Canada • Statistique Canada17
Responsive Collection Design (RCD) - ongoingFull RCD for SLID 2011 (including embedded experiment for 1st call)Improve current RCD strategy (e.g. propensity models, phase-in of
RCD, new conditions for decision making, cost-efficiency objective)RCD for CAPI surveysDocumentation
CATI cost-efficient framework (5 dimensions)Metrics used for costing and budgetingOptimal resources allocation within and between surveys (2)Collection process and practicesOperational constraints
Investigate approaches and assumptions to plan data collection for multi-mode surveys
Current and Future Research Plans 2..3
23-04-19Statistics Canada • Statistique Canada18
Paradata courseDescribe the paradata (e.g. type, contents, quality, etc.)Applications of paradata to plan, manage, monitor, assess and
improve the survey process Share experiencesLong and short versions
Other paradata research projectsSample coordination for CAPI surveysConsolidate and extend the use of audit TrailRCD - Theoretical frameworkSimulation and optimization projectsAd hoc research
Current and Future Research Plans 3..3
19 23-04-19Statistics Canada • Statistique Canada19
Potential Issues for Discussion
Are there important gaps in paradata research? If soWhich type of research need to be done? What are the research priorities?Any specific research with respect to TSE?
Sharing information (communication)Paradata working group, conferences/events (paradata sessions in
many international events), international network… Is it enough/too much? Is it efficient?
Potential collaboration between organizations - can it be improved?
What is the most efficient organizational structure for this type of research?
Statistics Canada • Statistique Canada 23-04-19
23-04-19Statistics Canada • Statistique Canada20
For more information, please contact
Pour plus d’information, veuillez contacter
François [email protected]