1. Questionnaire Design

• Any fixed set of questions administered to some group may be considered a questionnaire

• Need not be a probability sample, or a sample at all; may simply describe the group that fills out the questionnaire

The fundamental rule for designing good questionnaires (given voluntary participation)

• Maximize the perceived rewards of responding, and minimize the perceived costs, from the perspective of the respondent

• while satisfying your research objectives

Corollary

• Pretesting is always advisable (even if only on your spouse)

• Because, perceptions are what matter

2. Types of Questionnaires

Distinguish by methodology

• In person (including intercept)

• • Once upon a time, the gold standard

• By telephone

• • Mainstay of commercial market research

• By mail

• • Low response rates a problem

• By pick up/self-administered

• • E.g., hotel satisfaction cards

• By email/web site pop up

• • Incredible cost advantages; but nature of sample often an issue

3. Types of Questionnaires II

By content

• Heterogenous (typical)

• • Mix of classification data (e.g., demographics), behaviors, ratings of diverse topics, beliefs, etc.

• Opinions

• • Focus on agree/disagree opinion items and importance ratings

• Lifestyle

• • Focus on activities, interests, and opinions (generally for purposes of segmentation)

• Satisfaction

• • Focus on global and specific evaluations and on prioritization of same

• Readership

• • Classification questions combined with use of various media outlets

• Concept tests

4. Types of Questionnaires III

From the standpoint of good design, the differences between types are of secondary importance

• There is craft knowledge concerning specific ways to construct paper questionnaires versus interviewer questionnaires, for instance

• The capabilities of each methodthe types of responses that can be gatheredalso differ

• But, many principles of good design are relatively universal in applicability

5. Questionnaire Design Process

Generate potential topics, issues, items

• A group session may be helpful (leader should bring a seed to jump start discussion)

Construct a first draft, with actual questions, in a sequence,along with tentative answer categories

• Team leader generally undertakes

Vet against research questions & feasibility, redraft

Circulate 2 nddraft for comments

• Begin to finalize wording, answer categories, sequence

Generate penultimate draft, observing space constraints, and heeding all aspects of good design

Pretest

Finalize and administer

6. Content Generation for Questionnaires

Revisit your research objectives

• Generate specific topics and issues to reflect information required

Begin to generate specific questions

• Emphasis should be on closed-ended questions

• Make preliminary decision about answer categories: yes/no, multichotomous, ratings, etc.

• • Re-use categories from prior research whenever possible, to promote comparability

Phrasing questions, even at this early stage, is helpful in gaining perspective on whether crucial topics are being covered

Reviewing answer categories, with an eye to data analysis, similarly assists in judging whether the questionnaire will address the research question

7. Asking Good Questions I

Are these therightquestions?

• I.e., do the questions implement the research objective?

• Effective phrasing only matters if the correct topics and issues are addressed

Do the questions match the way the market works/the respondent thinks?

• If recall is not part of the purchase process, questions about what is recalled are irrelevant

• If consumers do not deliberate over the purchase, unpacking their deliberations makes no sense

• If a product purchase is contingent on some other purchase or event, asking about it in isolation wont help

8. Asking Good Questions II

Will respondentunderstandthe question?

Use simple words and simple syntax

• 8 thgrade reading level a good target

• Exception: technical audiences may require technical terms; but syntax should still be as simple as possible

Use words not open to varying interpretations

• What is your income is poorly phrased

• Be specific about who what when where

Does/can the respondent know the answer

• Not there, not involved

• Excessive memory load (restaurants)

9. Asking Good Questions III

Will respondents bewilling and ableto respond accurately?

Beware social desirability

Watch out for yea-saying bias

Avoid loaded questions

• Angel or devil words

• Extremes (always, never)

• Negative contractions

10. Asking Good Questions IV

Correct number of answer categories?

• 2 only (i.e., yes, no) often uncomfortable

• 3-4 often adequate (quality and frequency ratings)

• 5 appropriate for agree-disagree

• • Best used for opinions with no expected answer

• Not clear that larger numbers of categories outweigh disadvantages

• • Some tradition of 7 categories for semantic differential

• • Ten point scales have become familiar in the context ofevaluative judgments

Use only as many categories as needed

• Order responses on a single dimension

• Think carefully about dont know/NA categories

11. Asking Good Questions IV (contd)

Structure of answer category should reflect nature of response

Some responses have an intrinsic pro or con nature

• Agree disagree format, with 5 points, appropriate

Other responses are intrinsically none to lots

• E.g. frequency or amount

• Unidimensional scale labels required

Some responses have ordered, discrete categories

• Label each category

Other responses represent gradations on a continuum

• Need only label endpoints

12. Miscellaneous Guidelines

Put relevant, interesting, non-threatening questions first

Save generic questions (e.g., demographics) to last

Sensitive questions go in the late middle

Cluster related content to ease burden of responding

More white space is better

• A professional appearance can help a lot

Shorter is always better

13. A final bit of advice

Mock up the crucial tables that will be produced via analysis of answers

Now, looking at these mock-ups, how do you feel?

• Is this the information you need to make decisions?

• Are the answer categories exhaustive?

• Are there picayune distinctions that will be collapsed in most analyses?

A great way to identify superfluous questions and also missing questions.