Questionnaire Design Questionnaire Design MethodologyMethodology

1

ContentsContents

1. What are questionnaires? Why use them?

2. Properties of a questionnaire3. Procedures for questionnaire

design4. Principles of questionnaire design5. Questionnaire piloting6. Conclusions

2

Various definitions of Various definitions of questionnairequestionnaire• “ A set of questions arranged in a

certain structure to collect information from targeted group to answer one or several of research questions ”

• “A set of questions particularly used to address some statistically typical issues by gathering information in a survey.”

• And others…3

Why questionnaire?Why questionnaire?

Questionnaire is used to • Assess the need for investment and

development, and the differences among project targeted areas.

• To measure outcomes of project and intervention programmes

• Assess the beneficiaries’ satisfaction• Assess the project impacts

4

The importance of The importance of QuestionnaireQuestionnaireQuestionnaires are the most important

tool of a social investigationQuestionnaires reflect the research

issuesThe quality of a questionnaire

determines the quality of a surveyDesigning a high quality questionnaire

helps:◦Collect most accurate data◦ Increase response rate◦Reliable and valuable ◦Save time and money

Low quality Low quality questionnairesquestionnaires

Non response may affect the questionnaire validity just as same as missing data (or obtaining inaccurate data).

6

Low response ratesReduce the

effectiveness of sample

size Reduce the research power

Reduce the accuracy of final results

Procedures for questionnaire Procedures for questionnaire designdesign

3 major steps:1.Deciding on what to ask2.Defining types of question,

wording for each type of question3.Structuring and formatting the

questions

Deciding on what to askDeciding on what to ask

Define mid-level steps between the research question(s) and the final question(s)

Research question(s)Relevant mattersStay focusedQuestions

Defining types of questionDefining types of question

Yes/No questions >< Open questions

Single choice >< Multiple choiceCategorical questionsOrdinal questionsLinkert-scale questionsSingle-pole >< Double-pole

Wording for questionsWording for questions

Clear meaning, not vagueAvoid multi-meaning questionsAvoid negative questionsAvoided guiding questionsStay focusedQuestions

Structuring the questionsStructuring the questions

TitleIntroduction (enclosed with intro

letter)Contact Minimal types of questionAvoid horizontal & vertical

structureAvoid overuse of boxes, tables,

and lines

Structuring the questions Structuring the questions (cont.)(cont.)

Clear, complete and attractive structure

Two styles of question order:◦From easy to difficult, from general to

specific◦From most interesting to less interesting

No matter what style to be chosen, follow the logic, i.e. questions in each part are relevant

Keep the rhythm of the questionnaireAvoid overusing of skips and turns

The Art of asking The Art of asking questionsquestions• Ask the right question• Respondents must understand

the question• Respondents must know the

answer• Respondents must be willing and

able to give the answer

13

Types of Survey QuestionsTypes of Survey Questions

1. Questions about activities, facts/events

2. Questions about psychological status or attitudes

3. Questions about knowledge/perception

14

Questions with a specific Questions with a specific timetimeBad question:Over the past

6 months, what new household appliances have you purchased?

Better question:Now I’m going to read a

list of household devices. As I read each, please confirm whether you bought that appliance during the last 6 months.

How about…a refrigerator?

a kitchen device?a microwave?

15

Specifying…Specifying…Who, what, when, where and how?Ex. : Whose income? What are

included?Over what period of time? Ex.:

“In 2002, what was your total household income? Before tax? Please provide the incomes of each member of your family, including wages, social welfare, and public aids”

16

Questions with single-Questions with single-meaning wordsmeaning wordsNot clear:How do you

compare the distance between you and your family at the moment and in the past? Do you think you live closer to your family? Or still the same?

Clearer:Compared with

your previous house,

do you live closer now?

Or is it further? Or the same?

17

Open vs. Closed QuestionsOpen vs. Closed Questions

General rules: for quantitative research, closed questions are often better than open questions, because:◦Easier for respondents◦Easier for data entry, ◦Not necessary to recode data◦Easier for analysts to categorize

respondents

18

Responses to open & closed Responses to open & closed questionsquestions Open questions are those left open

for respondents to freely answer. “How do you feel about your health status today?”

Closed questions provide pre-determined answers“Do you feel well today?” Yes No

How much do you agree with the following statement: “Today my health is great”:

Agree Neither agree nor disagree Disagree

19

Closed formatClosed format

• Closed questions are often useful as it is…• Easy and quick to fill in• Easy to code, record and analyse

• Closed questions are often not useful as... • It does not cover all possible

responses (comprehensiveness) • It is not able to obtain deep

information

20

Disadvantages of Closed Disadvantages of Closed QuestionsQuestionsAnswers may lead the

respondentsIt may be too easy to answer

without much thinkingShould not be used when:

◦Asking for the frequency of sensitive behaviors

◦There are many different possible answers

21

Open formatOpen format

Open questions useful when…◦Answers are unknown◦Answers are too many/complicated to

be coded◦searching for further explanations of the

answers Open questions are not useful

when…◦There’s a need to code possible errors◦It is difficult to analyse and synthesize

22

Type of measuring scalesType of measuring scales Nominal scale

Ex. Male/female; Provinces: Bac Kan/Ha Tinh/Tra Vinh/Tuyen Quang…

Ordinal scale

Ex. Very poor – poor – average – rich – very rich; like – normal – dislike

Interval scale

Ex. range of income, range of age. Ratio scale

Ex. Percentage of male/female, ethnic groups

23

Format of response Format of response categories categories

Circling or tick off your answers

Skip & leave blank

24

Survey Intro/Cover LetterSurvey Intro/Cover Letter

Introduction should include:◦who is conducting the survey◦the topics covered in the survey◦Statement of confidentiality◦Statement that the time length of

the survey depends on the research mode, topics, population

25

Check the format of Check the format of questionnairequestionnaireNumber all the questions

sequentially Use large, clear font type;List groups of answers in vertical

sequenceAvoid double/triple-choice

answersBe consistent with guidelines and

orders of answer groups.

26

Check the format of Check the format of questionnairequestionnaireDon’t split question across pages. If necessary, move the question and

answers to next page.Put specific instructions of

questionnaire as needed, then the questions

Distinguish instructions from questions

Pre-code the questionnaire using check boxes

27

Principles for questionnaire Principles for questionnaire designdesignAsk what you want to attainAsk the way you want to gain the

answer: use of wordsUnderstand the mind of

respondents and the context of the interview

Other rules…

28

Use of wordsUse of wordsWording should be accurate: a

small change could make great differences

Words in a question affect the answer.

The more specific the question, the more it influences the answer.

29

Context of the interviewContext of the interviewMany cases are encountered:

◦The interviewer may even chitchat various topics with an enthusiastic respondent

◦In some cases, the interviewee may refuse to answer.

Notes for a new interviewer (see next page)

30

Rules for new interviewersRules for new interviewersAvoid jumping to the questions

until you really understand the research question

Once raising a question, ask yourself “why I need to know this matter?”

Use available questions from different sources.

31

Use available questionsUse available questionsReady-made questions from

previous successful investigations are encouraged to use, so that:◦To save time and money (short-cut)◦To compare with the previous results◦To improve the validity of the answers

of research with similar sample size, context and without changes

◦For investigations with different time and with changes, use the same questions to measure trend.

32

Use available questions Use available questions (cont.)(cont.)When using questions from other

surveys, be concerned with:◦Copyright◦Setting of question (as the answers

for some questions are particularly used for a certain situation)

33

Other principlesOther principlesGuarantee clear codes for main and

sub- questions (ex. 1, 2a, 3, 4b)Start with simpler questions (not

sensitive)Use proper logic. If the questionnaire is

long, start with the keysPut questions on anthropology at the

endPut questions for past experiences prior

to those for current status.Put general questions first, then specific

34

Other principles (cont.)Other principles (cont.)Apply the funnel-shaped rule

(from wide to narrow questions)Use the rule of filteringUse equivalent terms (avoid

technical ones)Avoid unclear terms/short termsAvoid multi-meaning words/termsKeep sentences short (if possible)

35

Other principles (cont.)Other principles (cont.)Avoid suggesting the answers (by

proposing a hypothesis)Avoid leading questionsBe cautious using hypothetical

questionsAvoid bias (see next)

36

Piloting: Testing the Piloting: Testing the QuestionnaireQuestionnairePreferable to test the

questionnaire with those similar to targeted subjects of the research population

Conduct pilot using the same mode for the major study

Consider cognitive pretesting

37

Piloting: Testing & Piloting: Testing & EvaluationEvaluation

38

Piloting 1. Develop &

test the research

tools

2. Assessing the feasibility of a survey

against sample size

3. Identify logistical problems of the questionnaire

Piloting: Testing & Piloting: Testing & EvaluationEvaluation

Prepare first draft and pre-testDistribute questionnaire to small sampleEvaluate responses & re-draft

Checklist items: (adapted by Bowling 2002) The questions measure what is intended wordings are understood by and means a

similar thing to all Instructions are easily understood/followed Do the responses cover all relevant issues? Any questions systematically/frequently

missed? Basically test for ‘bugs’

39

Complete the Questionnaire: Complete the Questionnaire: Questionnaire Design - Practical Questionnaire Design - Practical approachapproach

1. Decide what information is needed

2. Search for existing questions3. Focus groups4. Draft new questions/revise

existing ones5. Sequence the questions6. Get peer evaluation7. Revise and self-test or peer test

Complete the Questionnaire: Complete the Questionnaire: Questionnaire Design - Practical Questionnaire Design - Practical approachapproach

8. Think-aloud interviews9. Revise/eliminate questions10.Prepare interviewer instructions

for pilot test11.Pilot test (10-20 cases)12.Revise eliminate questions

based on respondent & interviewer comments

Complete the Questionnaire: Complete the Questionnaire: Questionnaire Design - Practical Questionnaire Design - Practical approachapproach

13.Pilot again, if necessary14.Prepare final interviewer instructions15.Modify questionnaires if interviewers

raise questions in training16.After the interview completes, debrief

interviewers for potential problems17.Use experience from a study for

planning a study in the future

ConclusionsConclusions

Not general recommendations, just issues to be aware of, consider and balance out

Consider the aims of study, population and resources available

Principal objective is to collect data that is RELIABLE, VALID & UNBIASED

43

Key References To Get Key References To Get StartedStartedDillman, Don. Mail and Internet Surveys:

The Tailored Design Method. New York: John, Wiley & Sons, Inc, 2000.

Bradburn, N, Sudman, S. and Wansink, B.. Asking Questions: The Definitive Guide to Questionnaire design – for Market Research, Political Polls, and Social and Health Questionnaires. San Francisco: Jossey Bass, 2004.

Oppenheim AN. Questionnaire design, interviewing and attitude measurement. 2nd ed. London: Pinter; 1992.See also questionnaires chapter in the handbook of survey

44