Survey Contents Creating an Effective Questionnaire.

Survey Contents

Creating an Effective Questionnaire

Survey Contents Devote time to design the questionnaire up front

—you cannot change your questions in mid-survey!

Remember that your goal is to create a valid and reliable instrument, accurately measuring the concepts in which you are interested.

Remember that your respondents will be “Lazy Thinkers.”

Survey Contents When creating questions, consider the

effects of the following elements: Type

Constraints, if any, placed on responses Wording

Clarity of questions and response options Order

Relative placement of questions in the instrument Content

Topics the questions cover

Survey Contents Type

Constraints, if any, placed on responses

Open-endedA survey question to which the respondent replies in

his or her own word, either by writing or by talking

Closed-ended or fixed-choiceA survey question that provides preformatted response

choices for the respondent to circle, check, mark, etc.



Open-ended Respondents are encouraged to give full reaction Anticipate amount of space one will need to give a

complete answer Try to limit to exploratory questions

Too many will tire subjects Lots of work to standardize False belief that you got the full answer

Most commonly used are specific such as, “What is your age in years?”



Closed-ended or fixed-choice Response options are limited For example:

Yes/no True/false Multiple choice with an “other” option

Race: White, Black, Hispanic, Asian, Other


Constraints, if any, placed on responses Closed-ended or fixed-choice:

Rating Scales Ask respondents to rate something like a person, place, thing,

idea, attitude, etc. on a numbered scale, often Likert scales Likert Scales, dominant form in GSS

1 2 3 4 5Strongly Agree Neither Disagree Strongly Agree A nor D Disagree

Semantic Differential a fifth grader is Bad GoodAwful ⊲ ⊲ ⊲ ⊲ ⊳ ⊳ ⊳ ⊳ Nice


Constraints, if any, placed on responses Closed-ended or fixed-choice:

Ranking Scales Force R’s to put a set of items in order For example: Place the following animals in order of

their ability to understand humans from lowest ability to highest ability.

Dog, Cat, Slug

Other types too


Constraints, if any, placed on responses Closed-ended or fixed-choice

Special Issue: Coding Assigning numbers to response options so computers

can operate with them Keep conventional logic and data analysis in mind when

determining your coding schemes for responses Code Dummy Variables with 0 = absence and 1 =

presence Increasing magnitude should be reflected by increasing

codes Remember to think of levels of measurement!!! The GSS is horrifically coded!



Closed-ended or fixed-choice Special Issue: Coding

If your variable were Religiosity, and you asked, “how religious do you consider yourself?”Then you should not have a coding scheme like: 1, very religious2, somewhat religious3, barely religious4, not at all religiousIt should be reversed!



Closed-ended or fixed-choice Special Issue: CodingFrom the general social survey codebook:

Do you Feel that you're constantly under stress?

CONTENT CODE SAMPLEYes 1 3681No 2 6422Don't know 8 51Not stated 9 595

This should be No = 0 and Yes =1 because the idea of the question is about stress. In binary logic, 1 equals the presence of something.



Closed-ended or fixed-choice Special Issue: CodingFrom the general social survey codebook:

How frequently do you participate in bowling?CONTENT CODE SAMPLE2-3/month 1 131-2/week 2 903+/week 3 5

This is actually appropriate, because the higher the code number, the more the person bowls.

Survey Contents Wording

Clarity of questions and response options

Respondents should understand your questions Pre-testing is a very effective way to see if they do A bad question:

Do you hate taxes? —What exactly are you measuring? The answer will tell you little because our society has many

different taxes, loves to complain about them, but thinks they are necessary

Bad words: AIDS prevention researchers learned to avoid words such

as “intercourse,” “homosexual sex,” and “intravenous injection” because many do not understand these words



Guidelines for good questions:1. Be direct2. Maintain simplicity3. Be specific4. Take the role of your respondent

The following points elaborate on these four main themes.



Some things to do to make questions less unclear and less incomprehensible:

Avoid complex rhetoric, syntax, or disciplinary slang or jargon.

Do not expect them to learn new words or information just to answer a question. (Sometimes, you may establish context with a short paragraph then ask a series of short, specific questions.)



Avoid Double-barreled questions. They contain two questions in one. For example, “Do you think that students and Professors

should be given discounts on sports tickets?”

Avoid Double-negative questions. For example, “Do you disagree that professors should not

be required to help students outside of class?”

Avoid hypothetical questions (unless you are studying the sociology of hypothetical situations). For example, “If men could have children, would your

husband or boyfriend stay with you?”



Avoid ambiguous questions.

“Do you teach your children to effectively function?” “Does your boss engage you in interactive dialogue?” “Effectively function” and “interactive dialogue” are subject to interpretation.

Words such as "usually" or "normally" mean different things to different people. “Do your customers normally complain?” has a variety of interpretations.



Avoid biased questions. Avoid making one response option look more suitable than the other, using emotionally loaded terms, or using unbalanced response categories.

“Don’t you think that suffering terminal cancer patients should be allowed to be released from their pain by choosing death?”

SJSU pays professors less than similar universities. SJSU should: 1. pay them more, 2. keep paying the same, 3. reduce pay little, 4. reduce pay some, 5. reduce pay a lot, 6. dramatically reduce pay



Avoid biased questions. Avoid making one response option look more suitable than the other, using emotionally loaded terms, or using unbalanced response categories. From the GSS:“What is your opinion about a married person having sexual

relations with someone other than the marriage partner –is it always wrong, almost always wrong, wrong only some times, or not wrong at all?”

Always wrong = 1 Almost always wrong = 2 Wrong only sometimes = 3 Not wrong at all = 4

Survey Contents Order

Relative placement of questions in the instrument

Opening questions should be simple and introduce the topic of the survey.

Try not to mix topics. Put like things into sets of questions.



Avoid framing later questions with topics that can be linked to them in previous questions.

For example, one should not ask about attitudes toward crack use right before asking about attitudes toward the urban poor. This will invoke stereotypes about the poor.



Sensitive questions should never be at the beginning. Put in middle. May need opening paragraph for them.

Consider need to transition between types of questions. Write transitory questions or directions.

Place your most important questions earlier—R’s may fatigue before answering them if they are later.



Do not put in too many complex format questions (e.g., “skipping” or “go to next section”). R’s will get confused/frustrated.

Closed-ended questions are easier to answer. Put open-ended questions later in survey. Investment of time up to that point will likely keep them involved through the open-ended.

Survey Contents Content


You should make the topic of the survey clear to R’s. Do not surprise them with questions about unrelated topics.

If including sensitive questions, provide a rationale for asking them. Explain how honest answers will be helpful to others.

Avoid phrasing questions to seem too personal or direct when dealing with culturally sensitive issues. For example, avoid “Do you abuse your kids?”

Questions relevant to deviance should include normalizing statements. For example, “Many people use drugs for a variety of reasons. Have you ever used ecstasy (or x, e, MDMA, Adam) to feel closer to other people?”

Survey Contents Content


The more sensitive the topic, the more reassurance of confidentiality should be stated.

Never open a survey with questions about sensitive topics, and do not end surveys with negative questions that will linger in R’s minds.

Continually reassure R’s that there are no right or wrong answers to questions—just truthful or not.

Questions that require lots of specific details or a good memory are typically useless. If they are necessary, employ techniques to prompt recollection.

Survey Contents When creating questions, consider the effects of

the following elements: Type





Now… Some General Advice

Survey Contents General Advice

Always consult other surveys first They will give you good ideas for exploring your topic They may provide good examples of how to ask

questions You avoid “reinventing the wheel”

Always pre-test your instrument

Survey Contents General Advice

When constructing and editing your survey, ask yourself:

Are your questions measuring what you are interested in?

Is every question integral to your research? (Do not waste R’s time.)

Do any questions seem to appear out of nowhere? Is each question general enough to cover the entire

sample’s experiences, but specific enough to get useful data?

Would you respond to these questions?

Survey ContentsSocial Psychology of Survey ResponseConsider these issues:

Not everyone will have an opinion on every topic. Consider “no opinion” as an option.

People want to appear to agree. Consider negative and positive statements.

R’s may lose track and choose salient options (such as first in series). Keep questions simple.

Response set problem: In a series of questions with the same response options, R’s may just quickly check the same response for all questions.


Bad memory leads to: Forward telescoping (reporting that events occurred

more recently than in reality) Backward telescoping (reporting events further back

than in reality). Salient events are over-reported Mundane events are underreported “Habitual” events will fill in for lost information.

Provide aides to recall such as reference points, landmark events, etc. Use limited time frames in questions. Such as: “During the last week, how often…”


Fatigue effect. Boredom or tiring. Use transitions, vary questions or response options, place easier questions near the end, or use a “jazzed up” format.

Self bias. People try to assert a vision of themselves and are the center of their universe. Try to elicit factual information rather than impressions by avoiding personal evaluations and by disguising implications about persons.

Survey ContentsSurvey Questions Assignment—Due next class:

Evaluate your course paper General Social Survey questions about by responding to each item below. Turn in a copy of your questions with this assignment.

1. Are the questions and response options effective at prompting accurate measures of social phenomena? How?

2. How might you change the questions to make them better, if at all?

3. How should the questions be recoded, if at all? Why?4. Are they sensitive questions that require caution on the

researchers’ part?5. Where would they best be placed in a survey?

Survey Contents Creating an Effective Questionnaire.

Documents

Transcript of Survey Contents Creating an Effective Questionnaire.