Innovations in online data collection for scientific ...€¦ · Data Quality (LISS panel) Several...
Transcript of Innovations in online data collection for scientific ...€¦ · Data Quality (LISS panel) Several...
![Page 1: Innovations in online data collection for scientific ...€¦ · Data Quality (LISS panel) Several studies, e.g. Scherpenzeel (2009) and Revilla and Saris (2010) Data collected via](https://reader035.fdocuments.net/reader035/viewer/2022070806/5f044c0e7e708231d40d478c/html5/thumbnails/1.jpg)
Innovation in online longitudinal data
collection for scientific research:
the Dutch LISS panel
Marcel Das
Workshop on Best Practices for EU-SILC Revision
Lisbon,15 October 2014
![Page 2: Innovations in online data collection for scientific ...€¦ · Data Quality (LISS panel) Several studies, e.g. Scherpenzeel (2009) and Revilla and Saris (2010) Data collected via](https://reader035.fdocuments.net/reader035/viewer/2022070806/5f044c0e7e708231d40d478c/html5/thumbnails/2.jpg)
Background
2
9-10-2014
Internet interviewing is a cost-efficient alternative to the traditional modes (Dillman and Bowker, 2001)
but there is more …
- maximal opportunities for innovation
- fast: data become available for analysis much more quickly
- technology allows for experimentation, (easy) follow-up data collection, and feedback from respondents
![Page 3: Innovations in online data collection for scientific ...€¦ · Data Quality (LISS panel) Several studies, e.g. Scherpenzeel (2009) and Revilla and Saris (2010) Data collected via](https://reader035.fdocuments.net/reader035/viewer/2022070806/5f044c0e7e708231d40d478c/html5/thumbnails/3.jpg)
MESS Project: a highly advanced research
infrastructure for the Social Sciences
3
9-10-2014
Immigrant panel
Links with
administrative
data
Experiments
New
measurement
devices
Core study
LISS
Panel
Open to scientists worldwide
![Page 4: Innovations in online data collection for scientific ...€¦ · Data Quality (LISS panel) Several studies, e.g. Scherpenzeel (2009) and Revilla and Saris (2010) Data collected via](https://reader035.fdocuments.net/reader035/viewer/2022070806/5f044c0e7e708231d40d478c/html5/thumbnails/4.jpg)
4
Online panel of 5,000 households
8,000 individuals (>= 16 years)
Questionnaires each month, 30 min.
Incentive 15 euro an hour (average)
Central element MESS project:
LISS panel
9-10-2014
![Page 5: Innovations in online data collection for scientific ...€¦ · Data Quality (LISS panel) Several studies, e.g. Scherpenzeel (2009) and Revilla and Saris (2010) Data collected via](https://reader035.fdocuments.net/reader035/viewer/2022070806/5f044c0e7e708231d40d478c/html5/thumbnails/5.jpg)
9-10-2014
5
Why use online panels for
scientific research?
![Page 6: Innovations in online data collection for scientific ...€¦ · Data Quality (LISS panel) Several studies, e.g. Scherpenzeel (2009) and Revilla and Saris (2010) Data collected via](https://reader035.fdocuments.net/reader035/viewer/2022070806/5f044c0e7e708231d40d478c/html5/thumbnails/6.jpg)
9-10-2014
6
Disadvantages online
surveys
Noncoverage:
Persons without Internet are usually excluded
No probability sample:
Visitors of specific websites
Self-selection (interested, strong opinions, specific goal)
![Page 7: Innovations in online data collection for scientific ...€¦ · Data Quality (LISS panel) Several studies, e.g. Scherpenzeel (2009) and Revilla and Saris (2010) Data collected via](https://reader035.fdocuments.net/reader035/viewer/2022070806/5f044c0e7e708231d40d478c/html5/thumbnails/7.jpg)
7
Online interviews as method, but:
Probability sample drawn from address sampling frame of Statistics Netherlands
Includes households without Internet access (less than 15%): CentERdata provides equipment
Contacted by letter, telephone or visit
LISS panel
9-10-2014
![Page 8: Innovations in online data collection for scientific ...€¦ · Data Quality (LISS panel) Several studies, e.g. Scherpenzeel (2009) and Revilla and Saris (2010) Data collected via](https://reader035.fdocuments.net/reader035/viewer/2022070806/5f044c0e7e708231d40d478c/html5/thumbnails/8.jpg)
Households without
Internet
?
8
9-10-2014
![Page 9: Innovations in online data collection for scientific ...€¦ · Data Quality (LISS panel) Several studies, e.g. Scherpenzeel (2009) and Revilla and Saris (2010) Data collected via](https://reader035.fdocuments.net/reader035/viewer/2022070806/5f044c0e7e708231d40d478c/html5/thumbnails/9.jpg)
9
Small and silent
Only the most frequently used functions
Automatic maintenance, safety
Simple operation and readable screens
Installation and support
simPC
9-10-2014
![Page 10: Innovations in online data collection for scientific ...€¦ · Data Quality (LISS panel) Several studies, e.g. Scherpenzeel (2009) and Revilla and Saris (2010) Data collected via](https://reader035.fdocuments.net/reader035/viewer/2022070806/5f044c0e7e708231d40d478c/html5/thumbnails/10.jpg)
9-10-2014
10
Recruitment of LISS panel
![Page 11: Innovations in online data collection for scientific ...€¦ · Data Quality (LISS panel) Several studies, e.g. Scherpenzeel (2009) and Revilla and Saris (2010) Data collected via](https://reader035.fdocuments.net/reader035/viewer/2022070806/5f044c0e7e708231d40d478c/html5/thumbnails/11.jpg)
9-10-2014
11
Internet
interviews
Letter and
brochure
Short
interview
SimPC and
broadband
Households with
Internet
Households
without Internet
Confirmation
Panel
question
Recruitment
![Page 12: Innovations in online data collection for scientific ...€¦ · Data Quality (LISS panel) Several studies, e.g. Scherpenzeel (2009) and Revilla and Saris (2010) Data collected via](https://reader035.fdocuments.net/reader035/viewer/2022070806/5f044c0e7e708231d40d478c/html5/thumbnails/12.jpg)
9-10-2014
12
Internet
interviews
Letter and
brochure
Short
interview
Households with
Internet
Households
without Internet
Confirmation
Panel
question
Response
Primary
response
target: 80%
Secondary
response
target: 60%
Tertiary
response
target: 53%
SimPC and
broadband
![Page 13: Innovations in online data collection for scientific ...€¦ · Data Quality (LISS panel) Several studies, e.g. Scherpenzeel (2009) and Revilla and Saris (2010) Data collected via](https://reader035.fdocuments.net/reader035/viewer/2022070806/5f044c0e7e708231d40d478c/html5/thumbnails/13.jpg)
9-10-2014
13
to optimize factors in the recruitment:
- contact mode
- incentive amount
- timing of the incentive
- content of the advance letter
- timing of the panel participation request
Recruitment experiment (1)
![Page 14: Innovations in online data collection for scientific ...€¦ · Data Quality (LISS panel) Several studies, e.g. Scherpenzeel (2009) and Revilla and Saris (2010) Data collected via](https://reader035.fdocuments.net/reader035/viewer/2022070806/5f044c0e7e708231d40d478c/html5/thumbnails/14.jpg)
9-10-2014
14
Experimental design CATI CAPI N
Special
letter
Standard
letter
Special
letter
Standard
letter
Prepaid
incentive
Introduce panel
in letter 0 euro 10 euro 200
Introduce panel
after interview 20 euro 50 euro 300
Promised
incentive
Introduce panel
in letter 0 euro 10 euro 400
Introduce panel
after interview 50 euro 20 euro 300
N 200 200 400 400 1200
Scherpenzeel, A. & Toepoel, V. (2012). “Recruiting a
Probability Sample for an Online Panel: Effects of
Contact Mode, Incentives, and Information,” Public
Opinion Quarterly 76(3), pp 470-490
Recruitment experiment (2)
![Page 15: Innovations in online data collection for scientific ...€¦ · Data Quality (LISS panel) Several studies, e.g. Scherpenzeel (2009) and Revilla and Saris (2010) Data collected via](https://reader035.fdocuments.net/reader035/viewer/2022070806/5f044c0e7e708231d40d478c/html5/thumbnails/15.jpg)
9-10-2014
15
Set up main recruitment
• Combination CATI – CAPI, follow up CAPI
• Prepaid 10 euro incentive
• Promised extra 10 euro incentive for starting
• Attention to design letter and brochure
![Page 16: Innovations in online data collection for scientific ...€¦ · Data Quality (LISS panel) Several studies, e.g. Scherpenzeel (2009) and Revilla and Saris (2010) Data collected via](https://reader035.fdocuments.net/reader035/viewer/2022070806/5f044c0e7e708231d40d478c/html5/thumbnails/16.jpg)
Response in recruitment
Recruitment stage (households)
% of total gross sample
LISS panel
Reached 90
Completed interview or central questions
75
Willing to participate in panel 63
Registered as panel member 48
Total gross sample* 9844
*excluding not usable addresses
9-10-2014
16
![Page 17: Innovations in online data collection for scientific ...€¦ · Data Quality (LISS panel) Several studies, e.g. Scherpenzeel (2009) and Revilla and Saris (2010) Data collected via](https://reader035.fdocuments.net/reader035/viewer/2022070806/5f044c0e7e708231d40d478c/html5/thumbnails/17.jpg)
9-10-2014
17
LISS panel compared to population (2008)
Group
LISS Panel %
Population %
Bias
Age >=70 7 13 -6
Living alone 15 20 -5
Non-western immigrant 4 7 -3
Did not vote at election 13 20 -7
Voted SP at election 17 17 0
Voted CDA at election 25 26 -1
Has Internet access 93 85 +8
![Page 18: Innovations in online data collection for scientific ...€¦ · Data Quality (LISS panel) Several studies, e.g. Scherpenzeel (2009) and Revilla and Saris (2010) Data collected via](https://reader035.fdocuments.net/reader035/viewer/2022070806/5f044c0e7e708231d40d478c/html5/thumbnails/18.jpg)
9-10-2014
18
LISS panel compared to DPES
LISS Panel
Bias DPES
Bias
Group
Age >=70 -6 1
Living alone -5 -4
Non-western immigrant -3 -2
Did not vote at election -7 -13
Voted SP at election 0 1
Voted CDA at election -1 1
Has Internet access 8 2
DPES = Dutch Parliamentary Electoral Studies
![Page 19: Innovations in online data collection for scientific ...€¦ · Data Quality (LISS panel) Several studies, e.g. Scherpenzeel (2009) and Revilla and Saris (2010) Data collected via](https://reader035.fdocuments.net/reader035/viewer/2022070806/5f044c0e7e708231d40d478c/html5/thumbnails/19.jpg)
19
Non-response patterns
Similar to those of other leading scientific panels
Superior to commercial access and volunteer panels: - no coverage problems - no self-selection
9-10-2014
![Page 20: Innovations in online data collection for scientific ...€¦ · Data Quality (LISS panel) Several studies, e.g. Scherpenzeel (2009) and Revilla and Saris (2010) Data collected via](https://reader035.fdocuments.net/reader035/viewer/2022070806/5f044c0e7e708231d40d478c/html5/thumbnails/20.jpg)
9-10-2014
20
Attention for difficult groups
Correction by refreshment sample in 2009 (stratified), 2011 (random), 2013 (stratified) in close collaboration with Statistics Netherlands
Separate immigrant panel in 2011 (with Cross-cultural Psychology, Tilburg University and Statistics Netherlands)
![Page 21: Innovations in online data collection for scientific ...€¦ · Data Quality (LISS panel) Several studies, e.g. Scherpenzeel (2009) and Revilla and Saris (2010) Data collected via](https://reader035.fdocuments.net/reader035/viewer/2022070806/5f044c0e7e708231d40d478c/html5/thumbnails/21.jpg)
21
Attrition LISS households, 2008 - 2014
9-10-2014
![Page 22: Innovations in online data collection for scientific ...€¦ · Data Quality (LISS panel) Several studies, e.g. Scherpenzeel (2009) and Revilla and Saris (2010) Data collected via](https://reader035.fdocuments.net/reader035/viewer/2022070806/5f044c0e7e708231d40d478c/html5/thumbnails/22.jpg)
22
Attrition panel members, 2008 - 2014
9-10-2014
![Page 23: Innovations in online data collection for scientific ...€¦ · Data Quality (LISS panel) Several studies, e.g. Scherpenzeel (2009) and Revilla and Saris (2010) Data collected via](https://reader035.fdocuments.net/reader035/viewer/2022070806/5f044c0e7e708231d40d478c/html5/thumbnails/23.jpg)
Data Quality (LISS panel)
Several studies, e.g. Scherpenzeel (2009) and Revilla and Saris (2010)
Data collected via the Internet are at least as valid and reliable as those collected with traditional methods
Revilla and Saris compared ESS questions in the regular face-to-face survey with the same questions in the LISS panel
9-10-2014
23
![Page 24: Innovations in online data collection for scientific ...€¦ · Data Quality (LISS panel) Several studies, e.g. Scherpenzeel (2009) and Revilla and Saris (2010) Data collected via](https://reader035.fdocuments.net/reader035/viewer/2022070806/5f044c0e7e708231d40d478c/html5/thumbnails/24.jpg)
9-10-2014
24
Use of the LISS panel
![Page 25: Innovations in online data collection for scientific ...€¦ · Data Quality (LISS panel) Several studies, e.g. Scherpenzeel (2009) and Revilla and Saris (2010) Data collected via](https://reader035.fdocuments.net/reader035/viewer/2022070806/5f044c0e7e708231d40d478c/html5/thumbnails/25.jpg)
Use of infrastructure
1. Collection of new data; proposals can be submitted throughout the year 2006 – mid 2014: budget available for academic researchers to use the LISS panel
2. Use of existing data
Available data:
- longitudinal core study
- proposed studies
9-10-2014
25
![Page 26: Innovations in online data collection for scientific ...€¦ · Data Quality (LISS panel) Several studies, e.g. Scherpenzeel (2009) and Revilla and Saris (2010) Data collected via](https://reader035.fdocuments.net/reader035/viewer/2022070806/5f044c0e7e708231d40d478c/html5/thumbnails/26.jpg)
Innovation in data collection
Smartphones (Time Use, with the Netherlands Institute for Social Research; Mobility, with Twente University)
Accelerometers (international collaboration) Internet bathroom scales 26
9-10-2014
![Page 27: Innovations in online data collection for scientific ...€¦ · Data Quality (LISS panel) Several studies, e.g. Scherpenzeel (2009) and Revilla and Saris (2010) Data collected via](https://reader035.fdocuments.net/reader035/viewer/2022070806/5f044c0e7e708231d40d478c/html5/thumbnails/27.jpg)
Time use diary app
Developed by CentERdata
Can be used any time
Notifications (reminders)
Closed answers: codes for activities
Experience sampling: 3 popup questions on feelings, random moments
Design study: 12 months data collection
170 panel members per month, 2,000 total
Panel member completes two days
200 loan smartphones, lend for 7 days 27
9-10-2014
![Page 28: Innovations in online data collection for scientific ...€¦ · Data Quality (LISS panel) Several studies, e.g. Scherpenzeel (2009) and Revilla and Saris (2010) Data collected via](https://reader035.fdocuments.net/reader035/viewer/2022070806/5f044c0e7e708231d40d478c/html5/thumbnails/28.jpg)
More information
Report: Using smartphones in survey research: a multifunctional tool. Available at www.scp.nl/smartphone
Fernee, H. & Sonck, N. (2013): Is everyone able to use a smartphone in survey research? Survey Practice, 6(4)
9-10-2014
28
![Page 29: Innovations in online data collection for scientific ...€¦ · Data Quality (LISS panel) Several studies, e.g. Scherpenzeel (2009) and Revilla and Saris (2010) Data collected via](https://reader035.fdocuments.net/reader035/viewer/2022070806/5f044c0e7e708231d40d478c/html5/thumbnails/29.jpg)
Mobility App
Developed by Novay (The Netherlands)
Uses GPS and Mobile Networks
Algorithm: distance, mode (speed), time
Webinterface: respondents check and adapt
Privacy issue: Consent procedure
Design study 3 waves spread over 3 years
500 panel members per wave
4 - 6 weeks measurement per wave
200 loan smartphones, lend for 4 weeks 29
9-10-2014
![Page 30: Innovations in online data collection for scientific ...€¦ · Data Quality (LISS panel) Several studies, e.g. Scherpenzeel (2009) and Revilla and Saris (2010) Data collected via](https://reader035.fdocuments.net/reader035/viewer/2022070806/5f044c0e7e708231d40d478c/html5/thumbnails/30.jpg)
Accelerometer study
Developed by Geneactiv (UK)
Measures acceleration, body temperature, day light
Design study: 13 weeks data collection
70 - 90 panel members per week, about 1,000 in total
Panel member wears 8 days
300 accelerometers, lend for 8 days
30
9-10-2014
![Page 31: Innovations in online data collection for scientific ...€¦ · Data Quality (LISS panel) Several studies, e.g. Scherpenzeel (2009) and Revilla and Saris (2010) Data collected via](https://reader035.fdocuments.net/reader035/viewer/2022070806/5f044c0e7e708231d40d478c/html5/thumbnails/31.jpg)
Bathroom scale study
Wireless bathroom scales
Uses radio signals through “bridge” and Internet, to send the data
Measures weight and body fat
Variability over time: more accurate and more frequent
Design study: Longitudinal study (approx. 3 years)
950 scales, last year 300 new scales added
Lend to household for at least 1 year
31
9-10-2014
![Page 32: Innovations in online data collection for scientific ...€¦ · Data Quality (LISS panel) Several studies, e.g. Scherpenzeel (2009) and Revilla and Saris (2010) Data collected via](https://reader035.fdocuments.net/reader035/viewer/2022070806/5f044c0e7e708231d40d478c/html5/thumbnails/32.jpg)
Results
Kooreman, P. & Scherpenzeel, A. (2014): High frequency body mass measurement, feedback, and health behaviors, Economics & Human Biology, 14
Highlights • Weight is 0.2 kilogram lower on Fridays than on Mondays;
BMI is 0.06 less and fat percentage is 0.03 lower on Fridays.
• The fat-based measure of obesity indicates a three times larger prevalence of obesity (53%) than the BMI-based measure (17%).
• A feedback that includes a recommended weight range increases the temporal variation in individual body weight by about 10%.
9-10-2014
32
![Page 33: Innovations in online data collection for scientific ...€¦ · Data Quality (LISS panel) Several studies, e.g. Scherpenzeel (2009) and Revilla and Saris (2010) Data collected via](https://reader035.fdocuments.net/reader035/viewer/2022070806/5f044c0e7e708231d40d478c/html5/thumbnails/33.jpg)
33
2. Use of existing data Who can use the LISS data:
every researcher who wants to use data for scientific, policy or societal relevant research
Use of data is free of charge (unless data are used for research funded by means of external sources)
Available data:
longitudinal core study
proposed studies 9-10-2014
![Page 34: Innovations in online data collection for scientific ...€¦ · Data Quality (LISS panel) Several studies, e.g. Scherpenzeel (2009) and Revilla and Saris (2010) Data collected via](https://reader035.fdocuments.net/reader035/viewer/2022070806/5f044c0e7e708231d40d478c/html5/thumbnails/34.jpg)
34
Longitudinal core study
Questions were designed in close collaboration with experts in the relevant fields
Core study borrows from various national and
international surveys Topics: Household and family, Economic
situation and housing, Work and schooling, Social integration and leisure, Health,
Personality, Religion and ethnicity, Politics and values
LISS wave 7: 2013 - 2014
9-10-2014
![Page 35: Innovations in online data collection for scientific ...€¦ · Data Quality (LISS panel) Several studies, e.g. Scherpenzeel (2009) and Revilla and Saris (2010) Data collected via](https://reader035.fdocuments.net/reader035/viewer/2022070806/5f044c0e7e708231d40d478c/html5/thumbnails/35.jpg)
LISS data
http://www.lissdata.nl
35
9-10-2014
![Page 36: Innovations in online data collection for scientific ...€¦ · Data Quality (LISS panel) Several studies, e.g. Scherpenzeel (2009) and Revilla and Saris (2010) Data collected via](https://reader035.fdocuments.net/reader035/viewer/2022070806/5f044c0e7e708231d40d478c/html5/thumbnails/36.jpg)
MESS is more cost-effective than other
approaches
For example, the LISS panel, at the core of MESS, is much less expensive than other major panels
Four times cheaper than PSID and HRS (in terms of cost per respondent per hour)
Two to three times cheaper than traditional surveys in the Netherlands
For the money spent, MESS generates more extensive data than traditional surveys
MESS also saves money by accommodating many new data collections at marginal cost 36
9-10-2014
![Page 37: Innovations in online data collection for scientific ...€¦ · Data Quality (LISS panel) Several studies, e.g. Scherpenzeel (2009) and Revilla and Saris (2010) Data collected via](https://reader035.fdocuments.net/reader035/viewer/2022070806/5f044c0e7e708231d40d478c/html5/thumbnails/37.jpg)
In its first phase, MESS has generated a
clear return on investment
An ultra-modern, efficient research infrastructure is now solidly in place
The LISS panel has collected more than seven years of rich and innovative data, using state-of-the-art methods
Researchers worldwide have accessed this data for use in scientific, policy, and societal studies
An innovative data archive is now easily available through a web tool on a dedicated LISS website (http://www.lissdata.nl)
9-10-2014
37
![Page 38: Innovations in online data collection for scientific ...€¦ · Data Quality (LISS panel) Several studies, e.g. Scherpenzeel (2009) and Revilla and Saris (2010) Data collected via](https://reader035.fdocuments.net/reader035/viewer/2022070806/5f044c0e7e708231d40d478c/html5/thumbnails/38.jpg)
Even higher return on investment…?
The return on investment will certainly grow over time, with an exponentially increasing number of publications (e.g. because of secondary analyses)
But also:
International collaboration!
This may lead to a wealth of international comparisons and new global data networks
9-10-2014
38
![Page 39: Innovations in online data collection for scientific ...€¦ · Data Quality (LISS panel) Several studies, e.g. Scherpenzeel (2009) and Revilla and Saris (2010) Data collected via](https://reader035.fdocuments.net/reader035/viewer/2022070806/5f044c0e7e708231d40d478c/html5/thumbnails/39.jpg)
www.lissdata.nl