Survey Design Webinar: Proven Methodologies for Surveys That Work

Greg Timpany Anup Surendran

Vice President, Product & EngineeringSenior Research Director, Global Knowledge; Partner with Anova Market Research

#SurveyDesignTips

What’s on the agenda today?

● How to Identify the Need

● Why do You Want to Run a Survey

● How to Design a Survey that Does What You Want

● Turning Goals Into Real Survey Questions

● Collecting Answers to Your Carefully Crafted Questions

● Turning Resources into Insightful and Actionable Data

All Is Not What You See

Photo by Uri Golman

The root cause of business issues often lie below the surface.

Management may see a decline in sales, but it is up to the researcher to isolate the cause.

Identifying the Need

● Research does not thrive if it is kept in a vacuum

● Look below the surface

● Query clients in order to understand what is known

It All Starts With a Plan

Source: Research Rockstar

Do we really need a survey?

Secondary Research

Pros● Can be less costly in terms of

time and budget● Leverages the work of other

analysts

Cons● It was designed to meet

someone else’s research needs and may not directly address your areas of interest

Primary Research

Pros● Directly designed to answer

your research questions● Provides you maximum

flexibility for analysis

Cons● Cost - primary projects

typically cost more than secondary research, in terms of time and budget

Why do you want to run a survey?

If secondary research including: online search, published reports, or an examination of existing in-house data will not allow you to answer your client’s questions then…

It is time for a survey!

Hey! What about qualitative research?QuantitativeQualitative

Designing a survey that works!

● Know your intended audience

● Know the overarching research questions

that need to be answered

● Balance the informational needs

● Respect the respondent’s time and

willingness to share their opinions

Survey Flow

Surveys have distinct blocks designed to guide the respondent through the process.

For example:

Respondents who do not meet screening criteria will be exited out.

Non-purchasers may be sent down one branch, while purchasers sent down another.

Judicious use of survey logic ensures that questions are appropriate for the type of respondent you are speaking to.

Survey Blocks

● Introduction:○ Survey duration - give respondent’s a realistic time estimate○ Remind them their opinions are secure

● Screeners: ○ Typical screeners include demographics (age, gender, region), level of

responsibility or purchase ability (B2B), product/service usage, or brand/company awareness

○ Screeners are useful for establishing quotas● Body● Demographics● Final Thoughts● Redirects

Question Types - Single Select

Multiple Choice (Single Select):

Asks respondents to select a single category. This is a go to question type for survey designers.

Keynotes:

● Keep category lists as brief as possible, preferably 5 - 9 items.

● Randomize to prevent order bias, unless your list is alphabetical, e.g. states, IT certifications, etc. or ranged, e.g. age and income.

● Can use drop down lists.● Include “Other (please specify)” where

appropriate.

Question Types - Multiple ResponseMultiple Choice (Select all that apply):

Asks respondents to select all options that apply from a category list.

Keynotes:

● Keep category lists as brief as possible, preferably 5 - 9 items.

● Randomize to prevent order bias.● Remind respondents they can select all that apply.● Make sure to include categories for “None” and

“Other (please specify)”● Serves a good input question for other types such

as single select, rank order or constant sum.● Each category is its own question. The sum of the

number selected can serve as a segmentation.

Question Types - Matrix or GridMatrix or Grid:

Useful for Likert-style questions as well as ratings. Items for consideration are in the rows and rating scale in the columns.

Keynotes:

● Keep item lists as brief as possible, preferably 5 - 9 items. If you are rating more items then consider formatting the list into multiple grids.

● Randomize items (rows) to prevent order bias.● A/B test scale direction low to high vs. high to low. In the US we tend to think from low to high, but

that may not be the case in international settings.● Can be created side by side so both importance and satisfaction, for example, can be measured

concurrently.● Scaled questions, such as a feature’s perceived importance to the purchase decision, can be subjective.

Consider rank order, constant sum or Max-Diff question types.

Question Types - Matrix or Grid

Question Types - Rank OrderRank Order:

Requires that the respondent order their selections based on some metric such as preference.

Keynotes:

● Keep items to be ranked as brief as possible, preferably 5 - 9 items. Respondents may have difficulty ranking more than this number of items.

● Randomize items to prevent order bias.● Source data from a prior question, such as a

multiple response, can be used to feed a rank order.

● Is less subjective than scaled questions.● Creates ordinal data.● Can use drag and drop (not advised for mobile)

Question Types - Constant SumConstant Sum:Requires that the respondent allocate a series of points, typically adding up to 100, across items.

Keynotes:● Keep list as brief as possible, preferably 5 - 9

items. Respondents may have difficulty ranking more than this number of items.

● Randomize items to prevent order bias.● Source data from a prior question, such as a

multiple response, can be used to feed a constant sum.

● Works well when tied to a numeric question, such as amount spent during last shopping trip.

● Is less subjective than scaled questions.● Creates metric data.

Question Types - Image Choice

Selecting images:

Requires the respondent to select or rate an image or images.

Keynotes:

● Great for soliciting market feedback on corporate images, such as logos, or for providing a higher level of engagement using pictures that are common knowledge, such as a bowl of strawberries or a sports car.

● Ensure that your images are clear at various resolutions and can be seen on different devices.

Question Types - Open-EndsCapturing Text:

Requires respondents to share thoughts in their own words.

Keynotes:

● Comes in a few different flavors● Where to place Open-Ends ● Should they be “required”● Safe to use open-ended questions throughout the

survey, after you have gotten through screeners.

Question Types - Advanced Questions Keynotes:

● Customer satisfaction can be measured using the Net Promoter Score question.

● Pricing can be addressed with the Van Westendorp price sensitivity model (PSM).

● Discrete choice or max-diff use cases can be leveraged with the conjoint models.

● Other options include the ability to generate heat maps, useful for testing visual imagery, and the opportunity for respondents to rate auditory or video content.

Formatting Considerations

● Lists vs. Dropdowns

● Branded vs. Blind

● Number of Questions per Page

● Fonts

● Use of Space

● Text

Formatting Considerations - Mobile● Readability

○ use mid-length scales (3 - 5 points) and avoid the need to pinch or zoom

● Clickability

○ It’s easier to click with a mouse than with your thumb. Be generous when sizing response buttons

● Loadability

○ Keep your images smaller and use fewer of them.

You can use more questions per page to reduce the number of pages to be loaded.

Traps to Avoid

● Managing Time

● Leading the Respondent

● Double-barreled Questions

● Insensitivity

● Overlapping Categories

● Thinking Small

● It Doesn’t Apply

● Failure to Follow-Up

● Death By Grid

Thank YouContact Information For more info, please contact us at

Greg Timpanygreg@anovamarketresearch.com919-673-0795

Anup Surendrananup.surendran@questionpro.com512-590-3369

@questionpro