ARM Lecture 6

8/14/2019 ARM Lecture 6

1/99

LOGO

Construction ofQuestionnaire

Construction ofQuestionnaire

Lecture # 6
http://www.amazon.co.uk/gp/product/images/0826451764/ref=dp_image_0?ie=UTF8&n=266239&s=bookshttp://www.amazon.co.uk/gp/product/images/0826451764/ref=dp_image_0?ie=UTF8&n=266239&s=books


2/99

www.themegallery.com Company Logo


3/99


4/99

LOGO

What is questionnaire?What is questionnaire?



5/99

What is questionnaire?

A questionnaire is a researchinstrument consisting of a series ofquestions and other prompts for thepurpose of gathering information from

respondents.Although they are often designed for

statistical analysis of the responses,this is not always the case.

Remember, good questionnaires taketime to develop. They are not justquestions on a page.

Company Logo


6/99

Questionnaires are used in samplesurveys or censuses to elicit reportsof facts, attitudes, and othersubjective states.

The developments are beginning totransform survey questionnaireconstruction from an art to a

science.


7/99

What is not questionnaire?

A questionnaire is not just a list of

questions.

It is tempting to begin questionnaire designby drafting possible qns on the survey

topics.

That is obviously important, but not the right

place to start.

We must first think about the following

topics.

bj i f i i


8/99

Objectives of questionnaireconstruction?

To maximize the proportion ofsubjects answering ourquestionnaire - that is, the responserate.

To obtain accurate relevantinformation for our survey



9/99

History

In 1086 William the Conquerorsurveyed the wealth andlandholdings of England using astandard set of inquiries andcompiled the results in theDomesday Book.

The questionnaire was invented by

Sir Francis Galton: 1874

Company Logo
http://upload.wikimedia.org/wikipedia/en/c/c2/Francis_Galton2.jpghttp://upload.wikimedia.org/wikipedia/en/c/c2/Francis_Galton2.jpg


10/99


11/99

Questionnaires are familiar to mostpeople. Nearly everyone has had someexperience completing questionnaires andthey generally do not make people

apprehensive.Questionnaires reduce bias. There is

uniform question presentation and nomiddle-man bias. The researcher's own

opinions will not influence the respondentto answer questions in a certain manner.There are no verbal or visual clues toinfluence the respondent.



12/99

Questionnaires are less intrusivethan telephone or face-to-facesurveys. When a respondentreceives a questionnaire in the mail,he is free to complete thequestionnaire on his own time-table.Unlike other research methods, the

respondent is not interrupted by theresearch instrument.


Di d t Of W itt


13/99

Disadvantages Of WrittenQuestionnaires

Possibility of low response rates:Low response is the curse ofstatistical analysis. It candramatically lower our confidence inthe results. Response rates varywidely from one questionnaire toanother (10% - 90%), however,

well-designed studies consistentlyproduce high response rates.


14/99

Inability to probe responses.Questionnaires are structuredinstruments. They allow little flexibility tothe respondent with respect to responseformat. In essence, they often lose the

"flavor of the response" (i.e., respondentsoften want to qualify their answers). By allowing frequent space for comments, the researcher can

partially overcome this disadvantage. Comments are among

the most helpful of all the information on the questionnaire,

and they usually provide insightful information that wouldhave otherwise been lost.



15/99

Nearly ninety percent of all communicationis visual. Gestures and other visual cuesare not available with writtenquestionnaires. The lack of personal

contact will have different effectsdepending on the type of information beingrequested. A questionnaire requestingfactual information will probably not beaffected by the lack of personal contact. A

questionnaire probing sensitive issues orattitudes may be severely affected.


16/99

When returned questionnaires arrive in themail, it's natural to assume that therespondent is the same person you sentthe questionnaire to. This may not actually

be the case. Many times businessquestionnaires get handed to otheremployees for completion. Housewivessometimes respond for their husbands.Kids respond as a prank. For a variety of

reasons, the respondent may not be whoyou think it is. It is a confounding errorinherent in questionnaires.


17/99

Finally, questionnaires are simplynot suited for some people. Forexample, a written survey to agroup of poorly educated peoplemight not work because of readingskill problems. More frequently,people are turned off by written

questionnaires because of misuse.



18/99

LOGO

Framework for

construction ofquestionnaire

Framework for

construction ofquestionnaire



19/99

Clarify goals


Conversations with several of the


20/99

Conversations with several of thestakeholders



21/99

Focus your goals



22/99

Invite people to take survey



23/99

Analyze responses


Translating data into meaningful


24/99

Translating data into meaningfulinformation



25/99

Present Results



26/99

Celebrate your success



27/99

Steps for DevelopingQuestionnaires

Develop the Purpose

Create the Conceptual Framework

Write Questions

Design the Questionnaire


28/99

Proper construction of thequestionnaire is essential to its

success

Powell, Ronald R.

www.themegallery.com

Framework: Questionnaire


29/99

Framework: QuestionnaireConstruction

Specify WhatInformation

Will be Sought

Identify

Correct

Respondent

Determine

Method of

Administration

Question Structure Question Wording Question Sequencing

PRETEST

and

Revise

Questionnaire


30/99

30

Steps to design a questionnaire:

1. Write out the primary and secondary aimsof your study.

2. Write out concepts/information to becollected that relates to these aims.

3. Review the current literature to identifyalready validated questionnaires thatmeasure your specific area of interest.

4. Compose a draft of your questionnaire.

5. Revise the draft.6. Assemble the final questionnaire.

Questionnaire Design - General


31/99

Questionnaire Design - GeneralConsiderations

Most problems with questionnaire analysiscan be traced back to the design phase ofthe project. Well-defined goals are thebest way to assure a good questionnaire

design.When the goals of a study can be

expressed in a few clear and concisesentences, the design of the questionnairebecomes considerably easier.

The questionnaire is developed to directlyaddress the goals of the study.



32/99

One of the best ways to clarify yourstudy goals is to decide how youintend to use the information. Dothis before you begin designing thestudy. This sounds obvious, butmany researchers neglect this task.Why do research if the results will

not be used?



33/99


34/99

As a general rule, with only a fewexceptions, long questionnaires get lessresponse than short questionnaires. Keepyour questionnaire short. In fact, theshorter the better.

Response rate is the single most importantindicator of how much confidence you canplace in the results. A low response rate canbe devastating to a study.

One of the most effective methods ofmaximizing response is to shorten thequestionnaire.



35/99

If your survey is over a few pages, try toeliminate questions.

Many people have difficulty knowing whichquestions could be eliminated.

For the elimination round, read eachquestion and ask, "How am I going to usethis information?" If the information willbe used in a decision-making process, then

keep the question... it's important. If not,throw it out.



36/99

One important way to assure a successfulsurvey is to include other experts andrelevant decision-makers in thequestionnaire design process.

Their suggestions will improve thequestionnaire and they will subsequentlyhave more confidence in the results.



37/99

Formulate a plan for doing the statisticalanalysis during the design stage of theproject. Know how every question will beanalyzed and be prepared to handle

missing data. If you cannot specify howyou intend to analyze a question or usethe information, do not use it in thesurvey.



38/99

Give your questionnaire a title that is short andmeaningful to the respondent. A questionnairewith a title is generally perceived to be morecredible than one without

Include clear and concise instructions on how

to complete the questionnaire. These must bevery easy to understand, so use shortsentences and basic vocabulary. Be sure toprint the return address on the questionnaireitself (since questionnaires often get separatedfrom the reply envelopes).



39/99

Begin with a few non-threatening andinteresting items. If the first items aretoo threatening or "boring", there islittle chance that the person will

complete the questionnaire.People generally look at the first few

questions before deciding whether ornot to complete the questionnaire. Make

them want to continue by puttinginteresting questions first.



40/99

Use simple and direct language. The questionsmust be clearly understood by the respondent.

The wording of a question should be simple andto the point.

Do not use uncommon words or long sentences.Make items as brief as possible. This will reducemisunderstandings and make the questionnaireappear easier to complete.

One way to eliminate misunderstandings is to

emphasize crucial words in each item by usingbold, italics or underlining.



41/99

Leave adequate space for respondents tomake comments.

One criticism of questionnaires is theirinability to retain the "flavor" of a

response. Leaving space for comments will provide

valuable information not captured by theresponse categories.

Leaving white space also makes thequestionnaire look easier and thisincreases response.



42/99

Place the most important items in the firsthalf of the questionnaire.

Respondents often send back partiallycompleted questionnaires.

By putting the most important items nearthe beginning, the partially completedquestionnaires will still contain importantinformation.



43/99

Use professional production methods forthe questionnaire--either desktoppublishing or typesetting and keylining.

Be creative.

Try different colored inks and paper. Theobject is to make your questionnaire standout from all the others the respondentreceives.



44/99

The final test of a questionnaire is to try it onrepresentatives of the target audience.

If there are problems with the questionnaire, theyalmost always show up here. If possible, be presentwhile a respondent is completing the questionnaire

and tell her that it is okay to ask you forclarification of any item.

The questions she asks are indicative of problems inthe questionnaire (i.e., the questions on thequestionnaire must be without any ambiguity

because there will be no chance to clarify aquestion when the survey is mailed).



45/99

LOGO

Types of questionsTypes of questions


Classification # 1


46/99

Classification # 1

1. Factual questions: age, gender

2. Opinion and attitude questions: intended todetermine persons ideas, inclinations, prejudices, with attitude scales, and indexes

3. Information questions: designed to measurerespondents knowledge

4. Self-perception questions: Similar to attitudequestions but are restricted to ones opinionsabout himself or herself

5. Standards of action questions: used to determinehow respondents would act in certaincircumstances

6. Questions about actual past or present behaviour:

7. Projective questions:

Classification # 2


47/99

Classification # 2

Four types of response scales forclosed-ended questions aredistinguished: Dichotomous, where the respondent has two options

Nominal-polytomous, where the respondent has morethan two unordered options

Ordinal-polytomous, where the respondent has more

than two ordered options

(bounded)Continuous, where the respondent ispresented with a continuous scale


48/99

1. Contingency questions - A question that is

answered only if the respondent gives aparticular response to a previous question.This avoids asking questions of people thatdo not apply to them

2. Matrix questions - Identical responsecategories are assigned to multiplequestions. The questions are placed oneunder the other, forming a matrix withresponse categories along the top and alist of questions down the side. This is anefficient use of page space andrespondents time.


49/99

3. Closed ended questions - Respondents answers are

limited to a fixed set of responses. Yes/no questions - The respondent answers with a yes or a

no.

Multiple choice - The respondent has several option from which

to choose.

Scaled questions - Responses are graded on a continuum(example : rate the appearance of the product on a scale from 1

to 10, with 10 being the most preferred appearance). Examples

of types of scales include the Likert scale, semantic differential

scale, and rank-order scale (See scale for a complete list of

scaling techniques.).


50/99

Question Structure


51/99

Question Structure

Open-end questions

If you were to purchase a new automobile during 2001, what brand of automobile wouldyou most likely purchase? ___________ brand

Closed-ended with ordered categories

If you were to purchase a new automobile during 2001, how likely is it that you would

purchase the following brands of automobiles?

Very Likely Somewhat Likely Somewhat Unlikely Very Unlikely

Toyota Camry 1 2 3 4Honda Accord 1 2 3 4

Nissan Maxima 1 2 3 4

Closed-ended with unordered categories

If you were to purchase a new automobile during 2001, which one of the following

automobiles would you most likely purchase?

Toyota Camry t Honda Accord tNissan Maxima t

Partially closed-ended

If you were to purchase a new automobile during 2001, which one of the following would

you most likely purchase?

Toyota Camry t Honda Accord t

Nissan Maxima t Other (please

specify)______________

Closed Vs Open ended Questions


52/99

Closed Vs. Open-ended Questions


Advantages DisadvantagesClosed-ended Easy and quick to answer

Answers across resps easy tocompareAnswers easier to analyze on

computerResponse choices make

question clearerEasy to replicate study

Can put ideas in resp's headResps w/ no opinion answer anywayResps can feel constrained/frustratedMany choices can be confusingCan't tell if resp. misinterpreted thequestion

Fine distinctions may be lostClerical mistakes easy to makeForce respondents into simple

responsesOpen-ended Permit unlimited number of

answersResps can qualify and clarify

responses

Can find the unanticipatedReveal resps thinking processes

Resps give answers w/ diff. level of

detailAnswers can be irrelevantInarticulate or forgetful resps are at

disadvantageCoding responses is subjective and

tediousRequires more resp. time and effortIntimidates respondentsWhen resp omits a response, can't tell

if its because

of belief or just forgetfulness

Closed Vs Open ended Questions


53/99

Closed Vs. Open-ended Questions

EvaluationCriteria

Advantages Disadvantages

Time Less time torespond & record

Longer time todevelop

Quick transfer tocomputer form



54/99

EvaluationCriteria


Cost Cheaper due to lower recording &interpreting time



55/99

EvaluationCriteria


Accuracy Less interviewer

recording error

Forcing answers into

categories

Respondentconvenience

More convenient toanswer



56/99

Structured Questions:


57/99

Response order effects

Oral presentation of response alternatives

Recency effectsexternally based processingeach read alternative terminates processing of previous alternative

increased processing of later response alternatives

QEffect on Respondent Answers

Source: An evaluation of a cognitive theory of response-order effects in survey

measurement, J. Krosnick and D. Alwin, Public Opinion Quarterly, 1987,

51, 1201-1219.

Question Sequence


58/99

Q qand Order Effects

QuestionDo you think a communist country likreporters come in and send back to t

Do you think the United States shoulother countries come in and send ba

Question sequence 2 YesDo you think the United States should let communist newspaper reportersfrom other countries come in and send back to their papers the news as theysee it?

Do you think a communist country like Russia should let American newspaperreporters come in and send back to their papers the news as they see it?

36%

66%

Once Committed

Hard to be

Unfair

37%

Difference !

Source: The Norm of Even-handedness in Surveys as in Life, H.

Schuman and J. Ludwig,American Sociological Review, 1983, 48,

Q ti i F t


59/99

Questionnaire Format

Matrix questions are often useful. They use space efficiently

Respondents can answer quickly

Respondents can compare across questions for consistency.

However, matrix questions have someproblems. You might be tempted to put questions into a matrix when

they belong in some other format.

Respondents may pattern their answers inappropriately.

Order effects may be quite pronounced, with strong potential

for abuse of the survey.


60/99

Some special comments on order effects. Early answers tend to influence later answers.

Some respondents (such as less educated respondents) are

more subject to order effects than other respondents.

Randomizing questions tends not to help - it just makes thesurvey more confusing.

Fun or interesting questions in the early part of a

questionnaire can maintain interest, but general,

demographic sorts of questions can establish rapport. You might want to adjust the order of questions according to

the type of survey interview or questionnaire.

Other Types


61/99

Other Types

Leading Questions: A leadingquestion is one that forces or impliesa certain type of answer. E.g. Superb

Excellent

Great

Good

Fair

Not so Great



62/99

Embarrassing Questions: Embarrassingquestions dealing with personal orprivate matters should be avoided. Yourdata is only as good as the trust and

care that your respondents give you. Ifyou make them feel uncomfortable, youwill lose their trust. Do not askembarrassing questions.



63/99

Hypothetical Questions:Hypothetical are based, at best, onconjecture and, at worst, on fantasy.I simple question such as: If you were governor, what would you do to combat

terrorism?

LOGO


64/99

LOGO

Most frequent errorsin questionnaires

Most frequent errorsin questionnaires


Issues to Consider
http://www.cc.gatech.edu/classes/cs6751_97_winter/Topics/quest-design/written.gifhttp://www.cc.gatech.edu/classes/cs6751_97_winter/Topics/quest-design/written.gif


65/99

To Neutral or Not to Neutral.

Think about how a respondent enters a response

scale and how many decisions the person must

make.

Strongly Disagree Disagree Neither Agree nor Disagree

Agree Strongly Agree

1 2 3 4 5


66/99

Pre-coding can save time with data entry.Has this presentation increased your understanding of questionnairedevelopment?

1Yes 2No

There is more than one way to ask a question. Use a

variety of question formats.

Pre-code


67/99


68/99


69/99

bad routing which leaves the interviewerwondering which question to ask next, orworse, routing to the wrong onetoo many questions

poorly laid out questions which are badly

grouped in the questionnaire

questions which have been missed outcompletely

pages of the questionnaire which are

missing or out of order.



70/99

Research showing that smallchanges in question wording ororder can substantially affectresponses has reinforced the

assumption that questions must beasked exactly as worded, and in thesame order, to produce comparabledata.


LOGO


71/99

LOGO

GuidelinesGuidelines


Quality aims in survey research


72/99

72

Qua ty a s su ey esea c

Goal is to collect information that is:

Valid: measures the quantity or concept thatis supposed to be measured

Reliable: measures the quantity or concept ina consistent or reproducible manner

Unbiased: measures the quantity or conceptin a way that does not systematically under-or overestimate the true value

Discriminating: can distinguish adequately

between respondents for whom the underlyinglevel of the quantity or concept is different

Qualities of a Good Question


73/99

Q Q

Evokes the truth. Questions must be non-threatening. When a respondent isconcerned about the consequences ofanswering a question in a particularmanner, there is a good possibility that theanswer will not be truthful. Anonymousquestionnaires that contain no identifyinginformation are more likely to producehonest responses than those identifying

the respondent. If your questionnaire doescontain sensitive items, be sure to clearlystate your policy on confidentiality.



74/99

Asks for an answer on only one dimension.The purpose of a survey is to find outinformation. A question that asks for aresponse on more than one dimension willnot provide the information you areseeking. For example, a researcherinvestigating a new food snack asks "Doyou like the texture and flavor of thesnack?" If a respondent answers "no",

then the researcher will not know if therespondent dislikes the texture or theflavor, or both



75/99


76/99

Has mutually exclusive options. A good

question leaves no ambiguity in the mind ofthe respondent. There should be only onecorrect or appropriate choice for therespondent to make. An obvious example is:

Where did you grow up?__ A. country

B. farm

C. city

A person who grew up on a farm in thecountry would not know whether to selectchoice A or B.



77/99


78/99

Order of questions: Go from general to particular.

Go from easy to difficult.

Go from factual to abstract.

Start with closed format questions.

Start with questions relevant to the main subject.

Do not start with demographic and personal

questions.



79/99

Follows comfortably from the previousquestion. Writing a questionnaire is similarto writing anything else. Transitionsbetween questions should be smooth.Grouping questions that are similar willmake the questionnaire easier tocomplete, and the respondent will feelmore comfortable. Questionnaires that

jump from one unrelated topic to another

feel disjointed and are not likely toproduce high response rates.

Company Logo


80/99

Does not presuppose a certain state ofaffairs. Among the most subtle mistakes inquestionnaire design are questions thatmake an unwarranted assumption. Anexample of this type of mistake is:

Are you satisfied with your current auto insurance? (Yes or No)

This question will present a problem forsomeone who does not currently have autoinsurance. Better:

Are you satisfied with your current auto insurance?

___Yes __No ___Don't have autoinsurance

Company Logo


81/99

Does not imply a desired answer. Thewording of a question is extremelyimportant. We are striving for objectivity inour surveys and, therefore, must be carefulnot to lead the respondent into giving the

answer we would like to receive. Leadingquestions are usually easily spottedbecause they use negative phraseology. Asexamples:

Wouldn't you like to receive our free brochure?

Don't you think the PPP Govt is spending too much money?



82/99

Does not use emotionally loaded orvaguely defined words. This is one of theareas overlooked by both beginners andexperienced researchers. Quantifyingadjectives (e.g., most, least, majority) arefrequently used in questions. It isimportant to understand that theseadjectives mean different things todifferent people.


83/99

Does not use unfamiliar words or

abbreviations. Remember who your audienceis and write your questionnaire for them. Donot use uncommon words or compoundsentences. Write short sentences.Abbreviations are okay if you are absolutely

certain that every single respondent willunderstand their meanings. If there is anydoubt at all, do not use the abbreviation. Thefollowing question might be okay if all the

respondents are accountants: What was your companys liquidity ratio last year?______

. How much did you spend last year for life insurance ? ______. How much did you spend last year for life insurance ? ______. How much did you spend last year for life insurance ? ______


84/99

Is not dependent on responses to previous

questions. Branching in written questionnairesshould be avoided. While branching can be used asan effective probing technique in telephone andface-to-face interviews, it should not be used inwritten questionnaires because it sometimesconfuses respondents. An example of branching is:

1. Do you currently have a life insurance policy ? (Yes or No) If no, go

to question 3

2. How much is your annual life insurance premium ? ______

Alternate Option:

How much did you spend last year for life insurance ? ______


85/99

Does not ask the respondent to order or rank a

series of more than five items. This becomesincreasingly difficult as the number of itemsincreases, and the answers become less reliable.This becomes especially problematic when askingrespondents to assign a percentage to a series of

items. In order to successfully complete this task,the respondent must mentally continue to re-adjust his answers until they total one hundredpercent. Limiting the number of items to five willmake it easier for the respondent to answer.


Questionnaire Construction


86/99

Keep it short, simple, and specific (KISSS)

Logical question progressionFlow more important than grammatical

correctness

Key questions first, sensitive last

Keep allied questions together

Be reasonable when requesting forecasts

Mix fully-structured and open-ended questions

Clear routing instructions, use colour

General Questionnaire Sequence


87/99

Start with simple more generalquestions nonthreatening

interesting

general respondent capability

Respondents will be looking for an

Excuseto Terminate or Stop Interview

Comment on this question?


88/99

q

How often did you talk to yourdental rep from XYZ during the pastthree months?

Never

Rarely

Occasionally

Regularly

Revision


89/99

e s o

How often did you talk to yourdental rep from XYZ during the pastthree months? Not at all

Once per month About 2-3 times per month

About once per week

More than once per week


90/99

Checklist for Question Wording


91/99

Q g

q vague - questions and answers

q bias or leadingq objectionable/embarrassingq too demanding

q check for double questionsq do they have the ability to

answer

q give frame of referenceq consider ability to compare

responses across past studies


92/99

Resources


93/99

Moser CA, Kalton G. Survey methods in socialinvestigation. 2nd ed. Aldershot: Gower; 1971.

Sheatsley, P.B. (1983) Questionnaireconstruction and item writing. In Ross, P.H.,Wright, J.D. and Anderson, A.B.(Eds.), Handbook

of Survey Research. New York: Academic Press.Sudman, S. and Bradburn, N.M. (1989)Asking

questions: A practical guide to questionnairedesign. San Francisco: Jossey Bass.



94/99

Oppenheim, A.N. (1992) Questionnairedesign, interviewing and attitudemeasurement. London: Pinter



95/99

http://www.marketresearchworld.netwww.statpac.com/surveys

Design and use of questionnaires:a review of best practice applicableto surveys of health service staffand patients, Health TechnologyAssessment, 2001. Vol.5, No. 31.

http://www.marketresearchworld.net/http://www.marketresearchworld.net/


96/99

Hot Tip


97/99


How do I incorporate my logo to aslide that will apply to all the otherslides?

On the [View] menu, point to [Master], andthen click [Slide Master] or[Notes Master].Change images to the one you like, then itwill apply to all the other slides.


98/99


99/99

ARM Lecture 6

Documents

Transcript of ARM Lecture 6