Do's and Don'ts of Dynamic Questions in Surveys

@FocusVisionInfo #NewMRX

Do’s and Don’ts of Dynamic Questions in Surveys Wednesday, August 26, 2015

Aaron Jue Michelle Andre Jamin Brazil

Part of the FocusVision “Quant+Qual” Webinar Series: A New World of Research with Q+Q

http://www.focusvision.com/



Today’s Agenda

1. What are dynamic questions?

2. Using dynamic questions in surveys…

3. Dynamic question experiments:- Mobile and Touch- Sliders

4. Dynamic question lesson learned: biases

5. Best practices and guidelines for using dynamic questions in your quantitative market research practice

@FocusVisionInfo#NewMRX

What are dynamic questions?

Radio Buttons Check boxes

Traditional survey forms: HTML-based question types…


Rank Sort

http://kb.decipherinc.com/index.php?View=entry&EntryID=1329


Slider

Buttons


Virtual Magazine

Image Map

Using dynamic questions in

your research…

• Dynamic questions allow you to gather different kinds of data and achieve deeper data capture through:

• Unique question types available: image map, dial test, shelf test…

• Improved respondent engagement

• Enhanced usability

@FocusVisionInfo#NewMRX

Respondent Engagement: These are fun!

Enhanced usability: Intuitive

Number Input VS Rank Sort

http://kb.decipherinc.com/index.php?View=entry&EntryID=1329

Enhanced usability: Intuitive

List VS Shelf test

Unlimited possibilities aren’t always a good thing…

Dynamic Questions

Experiments

What has research told us about how to use Dynamic Questions?

1. Experiment: Mobile and Touch

2. Experiment: Sliders

3. Big lesson learned

.

@FocusVisionInfo#newMRX

The mobile problem

Mobile share is growing, but mobile device show higher dropout rates in surveys…

Source: Decipher, Inc.

The mobile problem

Mobile survey design challenges: screen size

Source: Decipher, Inc.

Experiment: Mobile Touch Input Friendly

Not Mobile Optimized Mobile OptimizedVS

Experiment: Mobile Touch Input Friendly

Not Mobile Optimized Mobile Optimized (Card Sort)VS

Experiment: Mobile and Touch— Findings

Using dynamic questions to execute a mobile friendly survey can…

• Improve dropout rates by 50% among smartphone users

• Slightly reduce straight-line ticking

• Slightly reduce failure of trap questions

• Significant improvement in perceived survey enjoyment


Slight bias towards middle of the scale

Changing the fundamental structure of the question will alter response distribution


Tracker A Smartphone Tablet

Detractors -2% 0%Neutrals 4% 3%Promoters -2% -1%Mean 0.1 0.03

Gap (Optimized – Not Optimized)


Simply enlarging input area / improving readability doesn’t alter responses distributions…

Experiment: Sliders

Outside Start Left Start

Middle Start Right Start

1. Sliders and Traditional Radio Buttons2. Desktop, Smartphone, Tablet3. Vary start position of slider

Experiment: Sliders

For touch devices, mean scores are higher when slider starts on the right position

Experiment: Sliders

Mean Miles not significantly different

VS

Sliders may be suitable for continuous objective measures

Experiment: Sliders—Findings

Slider Findings• Responses are sensitive to start positioning especially for touch

devices• Data from objective measures appears less sensitive to slider biases• Though desktop users prefer traditional forms, touch input

respondents generally accepting of whichever scale type they encounter

• No difference in survey enjoyment

Slider Implications• Sliders could offer an alternative over text input• But otherwise tests didn’t show slider benefits over traditional forms,

and unexpected biases should give us pause

Dynamic questions

biases


The benefits of using dynamic questions are enormous, but customization can lead to unintended consequences.

Issues and bias are found with using dynamic questions…

With unlimited design possibilities, technology is moving faster than our understanding.

Multi-select bias

Checkbox Buttons

Bias: Fewer items selected

VS

Spanish

English

Cantonese

Tagalog

French

Color bias

Bias: Away from the negative end

VS

Design Dropouts

High respondent dropout

Cross-platform compatibility

What works on one platform may not be suitable for another

Which sport did you participate in most often?

Basketball

Football

Soccer

Volleyball

Track and Field

Tennis

Incrementally higher respondent dropout: Buttons are oversized for desktop view; saturation of color makes reading difficult

Dynamic Questions

Best Practices

• Focus on clear benefits

• Keep it obvious about what’s expected from your respondent

• Be consistent

• Test dynamic question functionality across devices


Focus on the benefit

Not Mobile Optimized Mobile OptimizedVS

Data collection: Dial test is not possible with traditional formats….


Usability Engagement

Pick list VS Shelf test


It’s cool…

.. “cool” can work but is fraught with potential pitfalls

Clarity

Make it apparent what to do: ideally no reading required

English

Spanish

Cantonese

Tagalog

French

Multi select??

Multi select!!

Consistency

Be consistent in how and when you use DQs• Sometimes being consistently wrong is better than having to be

absolutely right• A lot of the times research focuses on relative differences

Cross-platform functionality

BETTER

Ensure your DQ functions and renders well across devices

Which sport did you participate in most often?Basketball

FootballSoccer

VolleyballTrack and Field

TennisNOT SO GOOD

@FocusVisionInfo #NewMRX

Questions?

Aaron Jue Michelle Andre Jamin Brazil




[email protected]@focusvision.com

[email protected]

mailto:[email protected]



Do's and Don'ts of Dynamic Questions in Surveys

Marketing

Transcript of Do's and Don'ts of Dynamic Questions in Surveys