Survey Design and Measurement Jeremy Kees, Ph.D..
-
Upload
merry-crawford -
Category
Documents
-
view
223 -
download
1
Transcript of Survey Design and Measurement Jeremy Kees, Ph.D..
![Page 1: Survey Design and Measurement Jeremy Kees, Ph.D..](https://reader036.fdocuments.net/reader036/viewer/2022062517/56649f335503460f94c505b3/html5/thumbnails/1.jpg)
Survey Design and Measurement
Jeremy Kees, Ph.D.
![Page 2: Survey Design and Measurement Jeremy Kees, Ph.D..](https://reader036.fdocuments.net/reader036/viewer/2022062517/56649f335503460f94c505b3/html5/thumbnails/2.jpg)
Some practical issues….
• Qualtrics Research Platform– Free you under VSB’s “site license”– Extremely user friendly, but also very
robust– www.qualtrics.com
![Page 3: Survey Design and Measurement Jeremy Kees, Ph.D..](https://reader036.fdocuments.net/reader036/viewer/2022062517/56649f335503460f94c505b3/html5/thumbnails/3.jpg)
Some practical issues….
• Amazon Mechanical Turk– The most inexpensive way to collect
consumer data– Extremely user friendly, but also very
robust– www.mturk.com
![Page 4: Survey Design and Measurement Jeremy Kees, Ph.D..](https://reader036.fdocuments.net/reader036/viewer/2022062517/56649f335503460f94c505b3/html5/thumbnails/4.jpg)
Online Survey (created by you and housed on Qualtrics’ server)
Create HIT (Human Intelligence Task) on Mturk•Description of your study and a (Qualtrics) link to it
Mturk workers (survey responders) “work” on your HIT (i.e., they take your survey)
Data is recorded by Qualtrics. Participants who complete the survey are given a code to input into Mturk. Those that enter a valid code, get paid.
Everyone is happy
![Page 5: Survey Design and Measurement Jeremy Kees, Ph.D..](https://reader036.fdocuments.net/reader036/viewer/2022062517/56649f335503460f94c505b3/html5/thumbnails/5.jpg)
The Mturk data I collected today….
• N=200
• Cost = $100
• Data collected in less than 1 hour
• Demographics– Mean age = 36
– 56% male
– 76% Caucasian
– 80% at least some college• 41% are college grads
– Median income = $35-50k
– Highly engaged!
![Page 6: Survey Design and Measurement Jeremy Kees, Ph.D..](https://reader036.fdocuments.net/reader036/viewer/2022062517/56649f335503460f94c505b3/html5/thumbnails/6.jpg)
Stages in theResearch Process
Determine Research Design
Analyze and Interpret the Data
Design Sample and Collect Data
Formulate Problem
Design Data Collection Method and Forms
Prepare the Research Report
![Page 7: Survey Design and Measurement Jeremy Kees, Ph.D..](https://reader036.fdocuments.net/reader036/viewer/2022062517/56649f335503460f94c505b3/html5/thumbnails/7.jpg)
Surveys / Questionnaires
• The most common measurement instrument when quantitative data is sought– Descriptive research– Experiments– Modeling– Etc….
![Page 8: Survey Design and Measurement Jeremy Kees, Ph.D..](https://reader036.fdocuments.net/reader036/viewer/2022062517/56649f335503460f94c505b3/html5/thumbnails/8.jpg)
Developing Surveys
• Good, well-specified research objectives lead to good surveys
• Research design dictates what types of questions should be used– Exploratory research = unstructured
script– Confirmatory research = structured
survey
![Page 9: Survey Design and Measurement Jeremy Kees, Ph.D..](https://reader036.fdocuments.net/reader036/viewer/2022062517/56649f335503460f94c505b3/html5/thumbnails/9.jpg)
Desirable Characteristics
• Brief
• Objective
• Specific
• Relevant
![Page 10: Survey Design and Measurement Jeremy Kees, Ph.D..](https://reader036.fdocuments.net/reader036/viewer/2022062517/56649f335503460f94c505b3/html5/thumbnails/10.jpg)
Survey Methods
• Usually should determine administration method prior to developing items– Can dictate what types of questions you
should ask
• Internet panels have become the most efficient and versatile method to collect data– Phone is still a viable option
– Mall intercepts can still be useful
– Mail/fax makes little sense anymore
![Page 11: Survey Design and Measurement Jeremy Kees, Ph.D..](https://reader036.fdocuments.net/reader036/viewer/2022062517/56649f335503460f94c505b3/html5/thumbnails/11.jpg)
Mall Intercepts vs. E-Panels
mall internet paneltests tests members
household size 2.8 2.9 3.0average age 40.5 39.2 37.2employed 71% 72% 69%white 86% 88% 89%male 20% 21% 15%college 40% 43% 46%
Correlation between Responses:
mall vs. internetinternet test/retest reliability
purchase intent .86 .94frequency .94 .97liking .85 .91price / value .90 .99
![Page 12: Survey Design and Measurement Jeremy Kees, Ph.D..](https://reader036.fdocuments.net/reader036/viewer/2022062517/56649f335503460f94c505b3/html5/thumbnails/12.jpg)
Jeff Miller and Alan Hogg “Internet vs. Telephone Data Collection” Burke White Paper series 2 (4) (www.burke.com). Also see Ashok Ranchhod and Fan Zhou “Comparing Respondents of E-Mail and Mail Surveys,” Marketing Intelligence & Planning 19 (2001), 254.
Internet PhoneTime survey took to administer 12.5 19.4 minutes
Upon completion, would respondent participate in future studies? 35% yes 26% yes
More experienced Internet Users x
Used rating scale extreme “endpoints”more frequently x
E-Panels vs. Phone
![Page 13: Survey Design and Measurement Jeremy Kees, Ph.D..](https://reader036.fdocuments.net/reader036/viewer/2022062517/56649f335503460f94c505b3/html5/thumbnails/13.jpg)
Types of Questions
• Screening Variables
• Independent Variables
• Dependent Variables
• Classification Variables– Segmentation– Moderators
• Attention Filters
![Page 14: Survey Design and Measurement Jeremy Kees, Ph.D..](https://reader036.fdocuments.net/reader036/viewer/2022062517/56649f335503460f94c505b3/html5/thumbnails/14.jpg)
Primary Data: Overview
• Types of Primary Data– Demographic / Socioeconomic Characteristics– Psychological / Lifestyle Characteristics– Attitudes / Opinions– Awareness / Knowledge– Intentions– Motivation– Behavior
• What, how much, where, when, how, who• Purchase behavior vs. use behavior
• E.g., --- basic hierarchy of effects models• Example (CWL Study)
![Page 15: Survey Design and Measurement Jeremy Kees, Ph.D..](https://reader036.fdocuments.net/reader036/viewer/2022062517/56649f335503460f94c505b3/html5/thumbnails/15.jpg)
QUESTION WORDING - General Guidelines • Use simple words and questions• Avoid ambiguous words and questions• Avoid leading questions---be objective• Avoid implicit alternatives• Avoid generalizations and estimates ---
Be specific • Avoid double-barreled questions
![Page 16: Survey Design and Measurement Jeremy Kees, Ph.D..](https://reader036.fdocuments.net/reader036/viewer/2022062517/56649f335503460f94c505b3/html5/thumbnails/16.jpg)
1. What is your income?
$10,000 or less………………….1
$10,000 to $25,000……………..2
$25,000 to $50,000……………..3
$50,000 to $75,000………..…….4
$75,000 to $100,000..……..…….5
$100,000 or more…………..……6
What is the problem and how would you revise the question?
![Page 17: Survey Design and Measurement Jeremy Kees, Ph.D..](https://reader036.fdocuments.net/reader036/viewer/2022062517/56649f335503460f94c505b3/html5/thumbnails/17.jpg)
4. Is the speed and efficiency of the drive-in teller services at your regular bank…..(READ CATEGORIES)
Very Satisfactory………………………4
Somewhat satisfactory…………………3
Somewhat unsatisfactory………………2
Very unsatisfactory……………….……1
![Page 18: Survey Design and Measurement Jeremy Kees, Ph.D..](https://reader036.fdocuments.net/reader036/viewer/2022062517/56649f335503460f94c505b3/html5/thumbnails/18.jpg)
Question Wording
• It is good practice to use scales whenever possible– Likert or semantic differential– Multi-item
![Page 19: Survey Design and Measurement Jeremy Kees, Ph.D..](https://reader036.fdocuments.net/reader036/viewer/2022062517/56649f335503460f94c505b3/html5/thumbnails/19.jpg)
Itemized Rating Scales
• The respondents are provided with a scale that has a number or brief description associated with each category.
• The categories are ordered in terms of scale position, and the respondents are required to select the specified category that best describes the object being rated.
• The commonly used itemized rating scales are the Likert and semantic differential
![Page 20: Survey Design and Measurement Jeremy Kees, Ph.D..](https://reader036.fdocuments.net/reader036/viewer/2022062517/56649f335503460f94c505b3/html5/thumbnails/20.jpg)
Types of Scales
• Nominal scales: those that use only labels
• Ordinal scales: those with which the researcher can rank-order the respondents or responses
• Interval scales: those in which the distance between each descriptor is equal
• Ratio scales: ones in which a true zero exists
![Page 21: Survey Design and Measurement Jeremy Kees, Ph.D..](https://reader036.fdocuments.net/reader036/viewer/2022062517/56649f335503460f94c505b3/html5/thumbnails/21.jpg)
21
IntervalPlease indicate how much you like each soft drink by checking the appropriate position on thescale: dislike like
a lot dislike like a lotCoke ____ ____ ____ ___Dr. Pepper ____ ____ ____ ___Mountain Dew ____ ____ ____ ___Pepsi ____ ____ ____ ___Seven Up ____ ____ ____ ___Sprite ____ ____ ____ ___
NominalWhich of the soft drinks in the following list do you like? (Check ALL that apply):
___Coke___Dr. Pepper___Mountain Dew___Pepsi___Seven Up___Sprite
OrdinalRank the soft drinks according to how much you like each (most preferred drink = 1, and least preferred drink = 6):
___Coke___Dr. Pepper___Mountain Dew___Pepsi___Seven Up___Sprite
RatioPlease divide 100 points among these soft drinksTo represent how much you like each:
___Coke___Dr. Pepper___Mountain Dew___Pepsi___Seven Up___Sprite100
Examples…
![Page 22: Survey Design and Measurement Jeremy Kees, Ph.D..](https://reader036.fdocuments.net/reader036/viewer/2022062517/56649f335503460f94c505b3/html5/thumbnails/22.jpg)
Itemized Rating Scales
• Likert Scales– requires the respondents to indicate a degree of
agreement or disagreement with each of a series of statements about the stimulus objects
Strongly Disagree Neither Agree Strongly
disagree agree nor agree
disagree
1. Wal-Mart sells high quality merchandise. 1 2X 3 4 5
2. Wal-Mart has poor in-store service.1 2X 3 4 5
3. I like to shop at Wal-Mart . 1 2 3X 4 5
4. Wal-Mart has low prices . 1 2 3X 4 5
![Page 23: Survey Design and Measurement Jeremy Kees, Ph.D..](https://reader036.fdocuments.net/reader036/viewer/2022062517/56649f335503460f94c505b3/html5/thumbnails/23.jpg)
Itemized Rating Scales
• Semantic Differential Scales– End points associated with bipolar labels that have
semantic meaning
SEARS IS:
Powerful --:--:--:--:-X-:--:--: Weak
Unreliable --:--:--:--:--:-X-:--: Reliable
Modern --:--:--:--:--:--:-X-: Old-fashioned
![Page 24: Survey Design and Measurement Jeremy Kees, Ph.D..](https://reader036.fdocuments.net/reader036/viewer/2022062517/56649f335503460f94c505b3/html5/thumbnails/24.jpg)
Decisions for Itemized Scales
• Number of scale items• More is better, but there is a diminishing return
around 11 points (Nunnally 1978)• 7-point scales are customary
• Enough to discriminate• Allows for a scale midpoint• Manageable
• Odd/even number of categories
• Forced vs. non-forced
![Page 25: Survey Design and Measurement Jeremy Kees, Ph.D..](https://reader036.fdocuments.net/reader036/viewer/2022062517/56649f335503460f94c505b3/html5/thumbnails/25.jpg)
Why Multi-Item Scales??
Construct Abstract Concept
“Unobservable”
“Latent”
“Psychological”**Single items are typically not
sufficient to assess unobservable constructs
![Page 26: Survey Design and Measurement Jeremy Kees, Ph.D..](https://reader036.fdocuments.net/reader036/viewer/2022062517/56649f335503460f94c505b3/html5/thumbnails/26.jpg)
Multi-Item Scales are More “Reliable”• True Score Test Theory
– All measures have1. “True” Score2. “Error” (Random and Systematic)
• Good measures minimize the systematic error component of the score
• Types of Reliability– Inter-Rater– Test-Retest– Internal Consistency (Cronbach’s Alpha)
![Page 27: Survey Design and Measurement Jeremy Kees, Ph.D..](https://reader036.fdocuments.net/reader036/viewer/2022062517/56649f335503460f94c505b3/html5/thumbnails/27.jpg)
Developing Sound Measures
Step 2:
Specify Domainof the Construct
Generate Sampleof Items
Collect Data
Purify Measure
Assess Validity
Step 1:
Step 3:
Step 4:
Step 5:
![Page 28: Survey Design and Measurement Jeremy Kees, Ph.D..](https://reader036.fdocuments.net/reader036/viewer/2022062517/56649f335503460f94c505b3/html5/thumbnails/28.jpg)
Question Sequencing
• After you have developed your measures, think about the order in which they should be asked
![Page 29: Survey Design and Measurement Jeremy Kees, Ph.D..](https://reader036.fdocuments.net/reader036/viewer/2022062517/56649f335503460f94c505b3/html5/thumbnails/29.jpg)
QUESTION SEQUENCING - General Guidelines
• Use (more) simple, interesting opening questions
• Use the funnel approach, asking broad questions first, and follow with more specific questions
• Carefully design branching questions– Skip/display logic
– Ask for classification information last• Place more difficult or sensitive
questions near the end
![Page 30: Survey Design and Measurement Jeremy Kees, Ph.D..](https://reader036.fdocuments.net/reader036/viewer/2022062517/56649f335503460f94c505b3/html5/thumbnails/30.jpg)
Question ordering #1
1 – EVALUATION OF FAT LEVEL OF A PRODUCT
2 – EVALUATION OF OVERALL PROD. NUTRITIOUSNESS
3 – EVALUATION OF OVERALL PRODUCT ATTITUDE AND INTENTIONS TO PURCHASE
Question ordering #2
1 – EVALUATION OF OVERALL PRODUCT ATTITUDE AND INTENTIONS TO PURCHASE
2 – EVALUATION OF LEVEL OF PROD. NUTRITIOUSNESS
3 – EVALUATION OF FAT LEVEL OF PRODUCT
QUESTION SEQUENCING - General Guidelines
![Page 31: Survey Design and Measurement Jeremy Kees, Ph.D..](https://reader036.fdocuments.net/reader036/viewer/2022062517/56649f335503460f94c505b3/html5/thumbnails/31.jpg)
FOP Labeling Study
• We were interested in consumer evaluations of:1. Facts Up Front
2. All On-Package Labeling
3. Front-of-Package Nutrition Info
Why was question sequencing critical??
![Page 32: Survey Design and Measurement Jeremy Kees, Ph.D..](https://reader036.fdocuments.net/reader036/viewer/2022062517/56649f335503460f94c505b3/html5/thumbnails/32.jpg)
Tips for Maximizing Participation
• Offer an incentive ($$$)• Importance/relevance of the research
project and its purpose• Completing the questionnaire will take
only a short time• Answers are anonymous or
confidential• Reminder 2-3 days after the initial ask
![Page 33: Survey Design and Measurement Jeremy Kees, Ph.D..](https://reader036.fdocuments.net/reader036/viewer/2022062517/56649f335503460f94c505b3/html5/thumbnails/33.jpg)
Attention Filters
• Always include an attention filter to ensure that you are getting “quality” respondents– Eliminate “click throughs”
![Page 34: Survey Design and Measurement Jeremy Kees, Ph.D..](https://reader036.fdocuments.net/reader036/viewer/2022062517/56649f335503460f94c505b3/html5/thumbnails/34.jpg)
Attention Filters (Case Study)
• Advertising Experiment– Very stringent screening criteria
• Total # that started the study = 15,458• Number that qualified = 870
– Incidence Rate (IR) = 5.6%
• Number that qualified and passed the attention screener = 451– 48% failed the attention filter!!!
• NOT GOOD, criticalmix!
![Page 35: Survey Design and Measurement Jeremy Kees, Ph.D..](https://reader036.fdocuments.net/reader036/viewer/2022062517/56649f335503460f94c505b3/html5/thumbnails/35.jpg)
“Easy” Attention Filter
![Page 36: Survey Design and Measurement Jeremy Kees, Ph.D..](https://reader036.fdocuments.net/reader036/viewer/2022062517/56649f335503460f94c505b3/html5/thumbnails/36.jpg)
“Difficult” Attention Filter
![Page 37: Survey Design and Measurement Jeremy Kees, Ph.D..](https://reader036.fdocuments.net/reader036/viewer/2022062517/56649f335503460f94c505b3/html5/thumbnails/37.jpg)
And finally, remember the golden rule….
Do unto your respondents as you would have them do unto you!!
![Page 38: Survey Design and Measurement Jeremy Kees, Ph.D..](https://reader036.fdocuments.net/reader036/viewer/2022062517/56649f335503460f94c505b3/html5/thumbnails/38.jpg)
Team Assignment #2• Refine your research questions
– Need to be clear, concise, and “testable”
• Based on your research questions1. Design 2 potential studies that could
address your research questions1. Explain the benefits and weaknesses of each
approach
2. Pick the “best” design and explain your decision
(Note: Don’t worry about measurement or sampling too much---you’ll have your chance to do that later)
![Page 39: Survey Design and Measurement Jeremy Kees, Ph.D..](https://reader036.fdocuments.net/reader036/viewer/2022062517/56649f335503460f94c505b3/html5/thumbnails/39.jpg)
Team Assignment #3
** Don’t start on this assignment until you’ve read Fowler (CH 6-7)
• Based on your research design1. Write a paragraph about what your
measurement instrument is supposed to accomplish
2. Make a list of what should be measured to accomplish the goals of the study
3. Develop your measurement instrument
![Page 40: Survey Design and Measurement Jeremy Kees, Ph.D..](https://reader036.fdocuments.net/reader036/viewer/2022062517/56649f335503460f94c505b3/html5/thumbnails/40.jpg)
Team Assignment #3
• Deliverables include:1. A very clean, polished version that you
could use to actually collect data• This means you will need to carefully think
through all of the issues we covered tonight (e.g., set-up, ordering, length, multi-item scales, etc.)
2. Intro paragraph and variable list (see previous slide)
(Note: Don’t worry about defining your sample--you’ll have your chance to do that next week)