Effects of Design in Web Surveys
description
Transcript of Effects of Design in Web Surveys
![Page 1: Effects of Design in Web Surveys](https://reader035.fdocuments.net/reader035/viewer/2022062521/56816837550346895dddf97b/html5/thumbnails/1.jpg)
![Page 2: Effects of Design in Web Surveys](https://reader035.fdocuments.net/reader035/viewer/2022062521/56816837550346895dddf97b/html5/thumbnails/2.jpg)
04/25/23
Effects of Design in Web Surveys
Vera ToepoelTilburg UniversityThe Netherlands
![Page 3: Effects of Design in Web Surveys](https://reader035.fdocuments.net/reader035/viewer/2022062521/56816837550346895dddf97b/html5/thumbnails/3.jpg)
04/25/23
3
CentERdata: Two Online Panels1. CentERpanel:l
• Exists for 17 years• 2000 households• Respondents fill out questionnaires every
weekOnline interviews as method, but:
Probability sample drawn from address sampling frame of Statistics Netherlands
Recruitment of new panel members address-based
Includes households without internet access (less than 20%): Equipment
![Page 4: Effects of Design in Web Surveys](https://reader035.fdocuments.net/reader035/viewer/2022062521/56816837550346895dddf97b/html5/thumbnails/4.jpg)
04/25/23
4
CentERdata: Two Online Panels2. LISS Panel• Grant from The Netherlands Organisation for
Scientific Research • 5000 households• Established in 2007 (we fielded 1st
questionnaire!)• Respondents fill out questionnaires every
monthOnline interviews as method, but:
Probability sample drawn from address sampling frame of Statistics Netherlands
Contacted by letter, telephone or visit Includes households without internet access (less
than 20%): Equipment
![Page 5: Effects of Design in Web Surveys](https://reader035.fdocuments.net/reader035/viewer/2022062521/56816837550346895dddf97b/html5/thumbnails/5.jpg)
04/25/23
5
1 item per screen
![Page 6: Effects of Design in Web Surveys](https://reader035.fdocuments.net/reader035/viewer/2022062521/56816837550346895dddf97b/html5/thumbnails/6.jpg)
04/25/23
6
4 items per screen
![Page 7: Effects of Design in Web Surveys](https://reader035.fdocuments.net/reader035/viewer/2022062521/56816837550346895dddf97b/html5/thumbnails/7.jpg)
04/25/23
7
10 items per screen
![Page 8: Effects of Design in Web Surveys](https://reader035.fdocuments.net/reader035/viewer/2022062521/56816837550346895dddf97b/html5/thumbnails/8.jpg)
04/25/23
8
![Page 9: Effects of Design in Web Surveys](https://reader035.fdocuments.net/reader035/viewer/2022062521/56816837550346895dddf97b/html5/thumbnails/9.jpg)
04/25/23
9
Answer categories
![Page 10: Effects of Design in Web Surveys](https://reader035.fdocuments.net/reader035/viewer/2022062521/56816837550346895dddf97b/html5/thumbnails/10.jpg)
04/25/23
10
Open-ended
![Page 11: Effects of Design in Web Surveys](https://reader035.fdocuments.net/reader035/viewer/2022062521/56816837550346895dddf97b/html5/thumbnails/11.jpg)
04/25/23
11
Vertical: positive to negative
![Page 12: Effects of Design in Web Surveys](https://reader035.fdocuments.net/reader035/viewer/2022062521/56816837550346895dddf97b/html5/thumbnails/12.jpg)
04/25/23
12
Horizontal
![Page 13: Effects of Design in Web Surveys](https://reader035.fdocuments.net/reader035/viewer/2022062521/56816837550346895dddf97b/html5/thumbnails/13.jpg)
04/25/23
13
Numbers 1 to 5
![Page 14: Effects of Design in Web Surveys](https://reader035.fdocuments.net/reader035/viewer/2022062521/56816837550346895dddf97b/html5/thumbnails/14.jpg)
04/25/23
14
Numbers 5 to 1
![Page 15: Effects of Design in Web Surveys](https://reader035.fdocuments.net/reader035/viewer/2022062521/56816837550346895dddf97b/html5/thumbnails/15.jpg)
04/25/23
15
Numbers 2 to -2
![Page 16: Effects of Design in Web Surveys](https://reader035.fdocuments.net/reader035/viewer/2022062521/56816837550346895dddf97b/html5/thumbnails/16.jpg)
04/25/23
16
Trained Respondents: Panel conditioning• Content (knowledge on topics)
– Prepare for future surveys– Develop attitudes
• Procedure (question-answering process)– Learn how to interpret questions– Answer strategically– Speed through the survey
![Page 17: Effects of Design in Web Surveys](https://reader035.fdocuments.net/reader035/viewer/2022062521/56816837550346895dddf97b/html5/thumbnails/17.jpg)
04/25/23
17
Procedure (answer process)
• Differences between trained and fresh respondents with regard to web survey design choices– Items per screen– Response category effects– Question layout
![Page 18: Effects of Design in Web Surveys](https://reader035.fdocuments.net/reader035/viewer/2022062521/56816837550346895dddf97b/html5/thumbnails/18.jpg)
04/25/23
18
Overall:
Difference in mean duration of the entire survey between panels: 436 seconds for the trained panel and 576 seconds for the fresh panel.
![Page 19: Effects of Design in Web Surveys](https://reader035.fdocuments.net/reader035/viewer/2022062521/56816837550346895dddf97b/html5/thumbnails/19.jpg)
04/25/23
19
Experiment 1: Items per screen
• Social Desirability Scale • 10 items• 3 different formats:
– 1 item per screen– 5 items per screen– 10 items per screen
![Page 20: Effects of Design in Web Surveys](https://reader035.fdocuments.net/reader035/viewer/2022062521/56816837550346895dddf97b/html5/thumbnails/20.jpg)
04/25/23
20
Experiment 1: Items per screen
• Trained respondents had higher inter-item correlations for multiple-item-per-screen formats.
• No significant difference in item non-response.
• Mean score of the Social Desirability Scale showed no evidence for social desirability bias.
• The mean duration to complete the ten social desirability items did not differ significantly between panels.
![Page 21: Effects of Design in Web Surveys](https://reader035.fdocuments.net/reader035/viewer/2022062521/56816837550346895dddf97b/html5/thumbnails/21.jpg)
04/25/23
21
Experiment 2: Answer Categories
![Page 22: Effects of Design in Web Surveys](https://reader035.fdocuments.net/reader035/viewer/2022062521/56816837550346895dddf97b/html5/thumbnails/22.jpg)
04/25/23
22
Experiment 2: Answer Categories
![Page 23: Effects of Design in Web Surveys](https://reader035.fdocuments.net/reader035/viewer/2022062521/56816837550346895dddf97b/html5/thumbnails/23.jpg)
04/25/23
23
Experiment 2: Answer Categories
• Category effect found• No difference in category effect between
trained and fresh respondents
![Page 24: Effects of Design in Web Surveys](https://reader035.fdocuments.net/reader035/viewer/2022062521/56816837550346895dddf97b/html5/thumbnails/24.jpg)
04/25/23
24
Experiment 3: Question LayoutQuestion: Overall, how would you rate the
quality of education in the Netherlands?Answer: 5-point scale Six formats:1. Reference format (decremental)2. Reverse scale: incremental3. Horizontal layout4. Add numbers 1 to 5 to verbal labels5. Add numbers 5 to 1 to verbal labels6. Add numbers 2 to -2 to verbal labels
![Page 25: Effects of Design in Web Surveys](https://reader035.fdocuments.net/reader035/viewer/2022062521/56816837550346895dddf97b/html5/thumbnails/25.jpg)
04/25/23
25
Experiment 3: Question Layout
1. Decremental vs. incremental: T+ F2. Vertical vs. horizontal layout: -3. No numbers vs. numbers 1 to 5:-4. Numbers 1 to 5 vs. numbers 5 to 1: T+F5. Numbers 5 to 1 vs. Numbers 2 to -2: T+F
• Trained respondents more easily selected one of the first options.
T=significant differences in Trained panelF=significant differences in Fresh panel
![Page 26: Effects of Design in Web Surveys](https://reader035.fdocuments.net/reader035/viewer/2022062521/56816837550346895dddf97b/html5/thumbnails/26.jpg)
04/25/23
26
Design Effects in Web Surveys: Comparing Trained and Fresh Respondents
• Overall little differences between trained and fresh respondents
• Trained respondents are somewhat more sensitive to satisficing:– Shorter completion times– Higher inter-item correlations for multiple-
items-per-screen formats– Select first response options more often
![Page 27: Effects of Design in Web Surveys](https://reader035.fdocuments.net/reader035/viewer/2022062521/56816837550346895dddf97b/html5/thumbnails/27.jpg)
04/25/23
27
Current and Future Research
• It has been little more than a decade since systematic research was begun on visual design effects in web surveys.
• In the last decade dozens of studies have been conducted
• It is now important that we begin to understand the importance of each of the visual effects
• Can we reduce visual effects by effective question writing?!
![Page 28: Effects of Design in Web Surveys](https://reader035.fdocuments.net/reader035/viewer/2022062521/56816837550346895dddf97b/html5/thumbnails/28.jpg)
04/25/23
28
Effective Question Writing
Tourangeau, Couper, and Conrad (POQ 2007) suggest there may be a hierarchy of features that respondents attend to:
Verbal language>numbers> visual cues
Question: Can the effects of visual layout be diminished through greater use of verbal language and numbers?
![Page 29: Effects of Design in Web Surveys](https://reader035.fdocuments.net/reader035/viewer/2022062521/56816837550346895dddf97b/html5/thumbnails/29.jpg)
04/25/23
29
Experiment 1: Visual Heuristics(joint with Don Dillman)
• Tourangeau, Couper, and Conrad (POQ 2004; 2007):1. Middle means typical: respondents will see the middle
option as the most typical2. Left and top means first: the leftmost or top option
will be seen as the ‘first’ in conceptual sense3. Near means related: options that are physically near
each other are expected to be related conceptually4. Up means good: the top option will be seen as the
most desirable5. Like means close: visually similar options will be seen
as closer conceptually
Experimental conditions: • Polar point or fully labeled scale• With or without numbers (1 to 5)
![Page 30: Effects of Design in Web Surveys](https://reader035.fdocuments.net/reader035/viewer/2022062521/56816837550346895dddf97b/html5/thumbnails/30.jpg)
04/25/23
30
Middle Means Typical
Fully labeled: even spacing
Fully labeled: uneven spacing
![Page 31: Effects of Design in Web Surveys](https://reader035.fdocuments.net/reader035/viewer/2022062521/56816837550346895dddf97b/html5/thumbnails/31.jpg)
04/25/23
31
Left and Top Means First
Fully labeled with color: consistent ordering
Fully labeled with color: inconsistent ordering
![Page 32: Effects of Design in Web Surveys](https://reader035.fdocuments.net/reader035/viewer/2022062521/56816837550346895dddf97b/html5/thumbnails/32.jpg)
04/25/23
32
Near Means Related
Polar point with numbers: separate screens
Polar point with numbers: single screen
![Page 33: Effects of Design in Web Surveys](https://reader035.fdocuments.net/reader035/viewer/2022062521/56816837550346895dddf97b/html5/thumbnails/33.jpg)
04/25/23
33
Up Means Good
Polar point with numbers: incremental
Polar point with numbers: decremental
![Page 34: Effects of Design in Web Surveys](https://reader035.fdocuments.net/reader035/viewer/2022062521/56816837550346895dddf97b/html5/thumbnails/34.jpg)
04/25/23
34
Like Means Close
Polar point
Polar point with color
![Page 35: Effects of Design in Web Surveys](https://reader035.fdocuments.net/reader035/viewer/2022062521/56816837550346895dddf97b/html5/thumbnails/35.jpg)
04/25/23
35
Like Means Close
Polar point with numbers (1 to 5)
Polar point with different numbers (-2 to 2)
![Page 36: Effects of Design in Web Surveys](https://reader035.fdocuments.net/reader035/viewer/2022062521/56816837550346895dddf97b/html5/thumbnails/36.jpg)
04/25/23
36
Labels, numbers and visual heuristics: is there a hierarchy?
1. Middle Means Typical
2.Left and Top Means First
3. Near Means Related
4. Up Means Good
5. Like Means Close
Effect heuristic?
no no yes yes yes
Numbers reduced effect?
yes yes Color: yesDif. #: no
Effect heuristic in fully labeled scales?
no no yes no no
![Page 37: Effects of Design in Web Surveys](https://reader035.fdocuments.net/reader035/viewer/2022062521/56816837550346895dddf97b/html5/thumbnails/37.jpg)
04/25/23
37
Experiment 2: Pictures in web surveys (joint with Mick Couper)Replicate study Couper, Tourangeau, and
Kenyon (POQ 2004)– 1. No Picture– 2. Low frequency picture– 3. High frequency picture
Add verbal instructions– A. No verbal instruction– B. Instruction to include both high and low
frequency instances– C. Instruction to include only low
frequency instances
![Page 38: Effects of Design in Web Surveys](https://reader035.fdocuments.net/reader035/viewer/2022062521/56816837550346895dddf97b/html5/thumbnails/38.jpg)
04/25/23
38
Low and High frequency picture
![Page 39: Effects of Design in Web Surveys](https://reader035.fdocuments.net/reader035/viewer/2022062521/56816837550346895dddf97b/html5/thumbnails/39.jpg)
04/25/23
39
Can verbal instructions reduce the effects of pictures?MANOVA
– main effect instructions lambda=.597, p<.0001
– main effect pictures lambda=.964, p<.0001
– interaction instructions*pictures lambda=.9691,p<.0001
– This suggests that while both the main effect and interaction are significant, instructions explain more of the variation in the answers than pictures!
![Page 40: Effects of Design in Web Surveys](https://reader035.fdocuments.net/reader035/viewer/2022062521/56816837550346895dddf97b/html5/thumbnails/40.jpg)
04/25/23
40
Future Research
How to reduce visual design effects in web surveys
![Page 41: Effects of Design in Web Surveys](https://reader035.fdocuments.net/reader035/viewer/2022062521/56816837550346895dddf97b/html5/thumbnails/41.jpg)
04/25/23
41
LISS data
Every researcher (irrespective of nationality) who wants to collect data for scientific, policy or societal relevant research can collect data via the LISS panel at no cost Proposals can be submitted through
www.lissdata.nl Existing data free available for academic
use longitudinal core studies proposed studies disseminated through www.lissdata.nl
![Page 42: Effects of Design in Web Surveys](https://reader035.fdocuments.net/reader035/viewer/2022062521/56816837550346895dddf97b/html5/thumbnails/42.jpg)