Constructing questions for interviews & questionnaires

Theory and practice in social research

Shahbaz Mehmood

Preface

• These slides include one theoretical framework for constructing questions and a small set of good and bad practices for constructing questions

• This is not only or complete set, but gives hints about what should you think when constructing questions.

Errors in gathering data through survey procedures are

• Responders’ failure to understand questions an intended

• A lack of effort or interest on the part of responders

• Respondents’ unwillingness to admit certain attitudes or behaviours

• The failure of respondents’ memory or comprehension process in the stressed condition of the interview

• Interviewers failures of various kinds (e.g. the tendency to change wording, failures in presentation procedures and the adoption of faulty recording process

Examples that illustrate the inadequate of many of the questions

• Factual questions sometimes elicit invalid answer: As simple as age

• Relationship between what respondents say and what they actually do in not always very strong: E.g. racism: Questions: Do you let a black person inside a bar? Do they really?

• Responders’ attitudes, beliefs, opinions, habits interests often seem to be extraordinary unstable

• Small changes in wording sometimes produce major changes in the distribution of responses

• Respondents commonly misinterpret questions

• Answer to earlier question can affect to later question

• Order of options changes the respondents: e.g. middle option

• Format of question: Open vs. close

• Respondents answer even if they don't know about the topic

• Cultural context affects

The principal assumptions that have defined the general orientation adopted by survey researcher in

the past• The researcher has clearly defined the topic about which information is

required

• Respondents have the information that researcher requires

• Respondents are able to access the required information under the conditions of the research situation

• Respondents can understand every question as intended

• Respondents are willing (or at least, can be motivated) to give the information

• Responses are more valid if they know why that is asked

• Answers are more valid if they are not suggested

• The research situation does not affect to the results

• The process does not affect the respondent

• All responses are meaningfully comparable

The traditional survey model

Researcher/Interviewer Respondent

A carefully standardised physical stimulus (i.e. question)

A response (i.e. answer) expressed in term of a standardised format provided by the researcher

The key issue: the comparability of aswer

• The reseacher must be clear about the nature of the information required and encode a request for this format

• The respondent must decode this request in the way the researcher intends it to be decoded

• The respondent must encode an answer that contains the information researcher has requested

• The reseacher must decode an answer as the respondent intendedd it to be encoded

Symbolic interactioinist theory

• Human beings interpret and define each other's actions. Not only simple stimulus-response

• Human beings can be objects of their own attention. Argue themselves, take pride themselves…

• Conscious social behavior is intentional behavior. Situation affects.

• Interpreting, planning and acting are ongoing process which begin anew at every stage of a social process

• In situation there is a role and person wants to hold on it.

• Always exists

• In brief: social situation is constantly negotiating a shared definition of the situation; taking one another's viewpoint into account; and interpreting another's behavior

A model of symbolic interactionist view of question answer behaviour

Encodes question, taking into account own purposes and presumptions/knowledge about the respondent, and perceptions of the respondent's presumptions/knowledge about self

Decodes question, taking into account own purposes and presumptions/knowledge about the interviewer, and perceptions of the interviewer's presumptions/knowledge about self

Encodes question, taking into account own purposes and presumptions/knowledge about the interviewer, and perceptions of the interviewer's presumptions/knowledge about self

Encodes question, taking into account own purposes and presumptions/knowledge about the respondent, and perceptions of the respondent's presumptions/knowledge about self

Interviewer Responder

The implication of the symbolic interaction theory for social research

• Roles

• Respondents try to find a mutually shared definition of the situation

• Interpretation of researcher acts

• Clues of what kind of information researcher wants

Other

• Multidimensionality. Nothing is one-dimensional

• Level of generality in the statements

• Level of generality in the topic

• Utterance frame: descriptive, explanatory or evaluative

Defining topic properly

Encodes question.

The need to clearly specify the information that is required. The assumption that the respondent have the required information. The assumption that the respondents can access the information

Decodes question

Encodes answerDecodes answer

Interviewer Responder

Researcher has clearly defined the required information

• Defined topic– Global vs. local (29)– Dimensions (economic, technical...)

• Is PL good thing: Marketing vs. Technical

– E.g. better social secure in cost of taxes

• Researcher has clear idea what kind of information is required

Responder have required information

• Hypothetical question => hypothetical Answer• Willingness to answer vs. information to answer• Opinions vs. facts

– Grounded opinions

– Serious opinions

• Responds are capable of verbalising information.– Not hurry, give time to answer

Formulating intelligible requests for information

Encodes question.

Formulating an intelligible request for information.

Defining key terms for respondents and virtues of simplicity and breity

Decodes question

Encodes answerDecodes answer

Interviewer Responder

The meaning of individual words

• Context specific meanings – Ambiguities words

– Different meanings

– Cultural decencies

• Relative difficulty of words – foreign words

• The operation of unintended nuances associated with apparently similar words– Forbid vs. allow

The meaning of concepts

• Lack of empirical references– How should be– Difficul, unclear, wide concepts

• Context decencies (often, usually etc.)

• Concrete: Not: often, quite often, seldom. Use 1,2,3,4,5

Structural complexities

• Number of words

• Less is better, but not complex ones

• Grammatical– Asking too much in one question– divide & conquer– not what you like this and that!

• Not too much negatives: Double is bad

Good practices

• Descriptive introduction: I'd like to describe…

• Explain before question – The addition of clauses, phrases and

instructions

• Anecdotes

• Education of the respondent affects

Contextual influences on respondents' interpretation of questions

Encodes question.

Encodes answerDecodes answer

Interviewer Responder

Decodes question

Contextual clues that influence respondents’ interpretations

Clues afforded by either the Question itself or its components

• Clues associated with the question– Some existing idea always

– Leading questions

• Clues in components– Additional phrases

– Note: Vegetables e.g. spinach not associated vegetarian in general as intended

Clues in response options.

• Approve or disapprove - not only approve in question

• Scale

• Pre-set response options as memory clues– May not remember other options

– Information range covered by the options.

– Who counts own averages?

– Abstract: no knowledge of normative levels

• Response biased with number of options– Respondent favor first when reads, last when hears - vary the order

– Let responder tell, if no-one match

Impact of preceding questions

• Influence associated with prior Q. Issues:– Does one have any knowledge?– How deep is knowledge or how thorough– How one interprets

• Psychological need for consistent– Need to stay in one opinion

• Even-handedness– Fair for all sides

Impact of preceding questions (cont’d)

• The impact of the previous answers

• The impact of the overall situation– Semantics of human mind.– Willingness to give information

Contextual influences on respondents' interpretation of questions

Encodes question.

Encodes answer

The kind of answer given is defined by the dimension of the response framework that is employed

Decodes answer

Interviewer Responder

Decodes question

Descriptive accounts

• How to describe the situation

• Perspectives

Level of social generality of responses

• You? Singular or plural?

Explanations

• Respondent can frame an explanation in many different ways

• Why did you do X– Causal antecedent - what caused to do X– Goal antecedent - purpose for X– Enablement factors - how possible– Causal consequences - what happened after X– The researcher's expectations

Evaluations

• Always relative• Standards necessary does not exists• Evaluative standard that are external to the

question– Pleasant/easy flight - pilot vs. passenger

• Evaluative standard that are built in to the question– Agree -disagree

Limitations of human memory

Limitations of human memory

• Long term memory problems– Nobody remembers all– Ask from present to past, not past to present.

• Short term memory problems– Simple questions

Filters

Filters

• Establishing the relevance of questions to respondent– Respondents tend to answer all the questions

• “I don't know enough”

– Don't know vs. don't have opinion

– Position of filters• Middle and don’t know are problematic

• Middle category at all

– subjective viewpoint to strongly (dis)agree• How strongly?

Reducing question threat

Reducing question threat

• Always some threat

• Biased

• Refuse

How?

• Casual, Do you happen to have…?

• Imputation of deviance, You know everyone does.

• Anonymous

• Lessen psychological immediacy of the Q (e.g. other instead self. Numeric coding of alternatives)

• Decrease specificity of the information called (e.g. broad response categories)

• Adopt knowing so that respondents have to confirm rather than volunteer

• 'Kinsey' straight at eyes and ask

• Adopt indirect so that respond give answer without knowing

• Place threatening Q at the end of series

How? (cont’d)

• Door in the face: Ask direct, if does not answer ask indirect/about e.g. salary.

• Ask long, don’t hesitate top repeat, give time and encourage to use time

• The definition of Q threat

Threat causes bias

• Topics that are desirable and over-reported– Be good citizen

– Be well informed and cultural person

– Have fulfilled of moral and social responsibilities

• Topics that are socially undesirable and under-reported– Illness and disabilities

– Illegal and contra-normative behavior

– Financial status

• Feeling of guilt or personal fears

• Threat associated with the nature of the relationship between interviewer and respond

– Social equity

– Fear of political or economical sanctions

Open vs. closed question debate

Open vs. closed question debate

• Coding responses to open questions

• Formulating response option for closed question

Evaluation of the use of open questions

• Open Q doesn’t suggest answer -- or does it?– Respondent’s use of "probes"

• Indicate level of knowledge– Assumptions

• will answer open Q if knows

• don't try to answer open Q if don't know

• wil answer closed question if don't know

• Answers indicate the salience of the topic in the responder's mind– Is there evidence?

• Avoid format effect– Not in the middle!

– Is it in the closed?

Evaluation of the use of open questions

• Allows complex motivation– Indicates more how respondent has interpreted it

– Indicate motivation that have influenced respondent's orientation to the topic

– Indicates the frame of reference

• Problems associated with probing inadequate answer – Turns to close

• Problems associated with coding response!

Evaluation assumptions associated with closed questions

• Answers the question in the same way -meaningfully comparable?

• Easier to answer • More easily analyzed

– Interpretations

• Problems associated with recording responses to closed questions– Start with general instructions

– Explain why to answer

Measuring attitudes

Measuring attitudes

• List of respond alternatives p. 153

• Define topic clearly

• Applicability of the topic to respondents has to be established– Don’t know

• Respondents has to know what sort of answers they should know– Level of generality

• Specs of standards– "Strongly agree"

Measuring attitudes

• Stimulus centered effects – Number of categories

• 7 +/- 2

– Anchoring effects of he category labels• The word in the positive/negative end?

• Problems in the batteries of rating scales– Meaning can alter– Some category

• long list. Don't know last when answering the first

• Ambiguity

Checks to ensure that questions work as intended

Checks to ensure that questions work as intended

• Editing rules (list p. 184)• Piloting Question

– Observation hard, but must be done in the beginning to ensure that questions work e.g. had to repeat?

• Question testing– Rephrase the question in responder's own words

– Double interview• Come in the beginning

– Allow aloud thinking

Esim KKK

• Kuinka monta parametroitua komponenttia on tyypillisessä toimitettavassa tuotteessa (0=ei käytetä)?

• Miten paljon parametreja on tyypillisessä parametroidussa komponentissa?

• Parametrien määrän vaihteluväli?

• Millaisia parametreja komponenteissa on?

• Millaisia arvoalueita parametreilla on (kokonaisluku, arvoalue, yksi kokonaisluku joukosta , yksi annetuista vaihtoehdoista, reaalilukuja, joukkoja)? Mitä muita?

Koponentti parametrinen vs komponetin mukanaolo parametrista

Tieto vai mielipideHalu vastata vai vastaus olemassa?

Mikä on komponentti?

Mikä on tyypillinen tuote?

Strukturaalinen kompleksisuus, liian monimutkainen lause:Kun toimitatte tuotteen, tuotteessa om komponentteja. Kompoenenteilla voi olla toimituskohtaisia parametreja. Onko olemassa tällaisia komponentteja, joissa on parametreja? Montako tälläistä kompoenttia, jossa on parametreja, tuotteessa on?

Vihjaa että pitäisi olla?

Jos ymmärsi aluksi väärin, halu konsistenttiin ilmaisuun