Surveying

Data collection methods

• Interviews• Focus groups• Surveys/Questionnaires

When we Use Surveys

• Requirements specification• User and task analysis• Prototype testing• User feedback

Surveys

• Principles, methods of survey research in general

• Content of surveys for needs and usability

• Survey methods for needs, usability

Definitions• Survey:

– (n): A gathering of a sample of data or opinions considered to be representative of a whole.

– (v): To conduct a statistical survey on. • Questionnaire: (n) A form containing a set of

questions, especially one addressed to a statistically significant number of subjects as a way of gathering information for a survey.

• Interview – (n): A conversation, such as one conducted by

a reporter, in which facts or statements are elicited from another.

– (v) To obtain an interview from. – American Heritage Dictionary

Surveying Steps

• Sample selection• Questionnaire construction• Data collection• Data analysis

Surveys – detailed steps

• Determine purpose, information needed• Identify target audience(s)• Select method of administration • Design sampling method• Design prelim questionnaire

– including analysis– Often based on unstructured or semi-

structured interviews with people like your respondents

• Pretest, revise• Administer: draw sample, administer q’aire,

follow-up non-respondents• Analyze results

Why surveys?

• Answers from many people, including those at a distance

• Relatively easy to administer• Can continue for a long time• Easy to analyze• Yield quantitative data

– Incl. Comparable x time

Ways of Administering Surveys

• In person• Phone • Mail• Paper, in person• Email (usually with a link)• Web

Possible Data

• Facts– Characteristics of respondents– Self-reported behavior

• This instance• Generally/usually• Past

• Opinions and attitudes:– Preferences, opinions, satisfaction, concerns

Some Limits of Surveys

• Reaching users easier than non-users, members/non-members, insiders/outsiders

• Relies on voluntary cooperation, possibly biasing responses

• Questions have to be unambiguous, amenable to short answers

• Self-reports• Only get answers to the questions you ask• The longer, more complex, more sensitive the

survey the less cooperation

Some sources of error

• Sample, respondents• Question choice• Question wording• Method of administration• Inferences from the data• Users’ interests in influencing results

– “vote and view the results”CNN quick vote: http://www.cnn.com/

When to do interviews?

• Need details that can’t get from survey• Need more open-ended discussions with

users• Small #s OK• Can identify and gain cooperation from

target group• Sometimes: want to influence

respondents as well as get info from them

Sample selection

Targeting respondents

• What info do you need?• From whom can you get the information

you need?– E.g. non-users are hard to reach– Can’t ask 5-year-olds; what do their parents

know?

Samples

• Probability samples – random selection– SimpleRandom Sampling– Stratified Random Sampling– Systematic Random Sampling– Cluster (Area) Random Sampling– Multi-Stage Sampling

• Non-probability – not random selection– Quota samples

• Proportional; nonproportional

– Convenience samples– Purposive samples– Snowballing

Sampling terminology

• Sampling Element: the unit about which info is collected; unit of analysis. E.g., members of households with access to the internet.

• Universe: hypothetical aggregation of all elements. All US households with access to the Internet.

• Population: theoretically specified aggregation of survey elements. I.e., next slide.

• Survey or study population: aggregate of elements from which the sample is actually selected. Households in US etc. etc. with phones…if a telephone survey.

Internet use

• A Nation Online:– Individuals age 3+– “Is there a computer or

laptop in this household?”– “Does anyone in this

household connect to the Internet from home?”

– “Other than a computer or laptop, does anyone in this household have some other device with which they can access the Internet, such as:

•  cellular phone or pager•  a personal digital assistant

or handheld device• a TV-based Internet device• something else/ specify”

– Sept. 2001: 143,000,000

• Nielsen//NetRatings – “all members (2 years of

age or older) of U.S. households which currently have access to the Internet.”

– “Internet usage estimates are based on a sample of households that have access to the Internet and use the following platforms: Windows 95/98/NT, and MacOS 8 or higher”

– Sept. 2001: 168,600,000• (+18%)

Terminology, cont.

• Sampling unit: element considered for selection. E.g., household. Census tracts and then households.

• Sampling frame: list of units composing population from which sample is selected. E.g, phone book

• Observation unit: unit from which data is collected. E.g. one person (observational unit) may be asked about the household or all members of the household. A person may be asked about a transaction or event.

• Sample: aggregation of elements actually included in study.

Terminology, cont.

• Variable: a set of mutually exclusive characteristics such as sex, age, frequency of use.

• Parameter: summary description of a given variable in a population.

• Statistics: summary description of a given variable in a sample.

Sample design

• Probability samples– random– stratified random– cluster– Systematic– GOAL: Representative sample

• Non-probability sampling– Convenience sampling – many web surveys– Purposive sampling– Quota sampling

Representative samples

• Which characteristics matter? • Want the sample to be roughly

proportional to the population in terms of groups/characteristics that matter– Exception: oversampling small groups

• E.g., students by gender and grad/undergrad status; students by major

Sample size

• Formulas for sample sizes are based on probability samples from very large populations– Size: if 10/90% split, 100; if 50/50, 400;

If a table, 30-50 in each cell

• To break down responses x groups, need large enough sample in each cell– Oversample small groups – e.g., Internet

use surveys and Hispanics– Later, correct for oversampling by weighting

in data analysis

Crosstabs

Undergrads(n=120)

%

Grads(n=200)

%

Total(n=320)

%

Satisfied 60%(71)

13%(25)

31%(96)

Dissatis. 40(47)

87(165)

69(127)

Total 100%n = 118

100%n = 190

100%n = 308

No ans. n = 2 n = 10 n= 12

Needs, usability, and sampling

• Requirements specification– Convenience sample of current users– Purposive sample of employees, users– Quota sample

• E.g., x from each location, department

• Prototype evaluation– Questionnaire as a way of getting

consistent data from test population – probably in entirety; but could be any of the above

• User feedback– User surveys; comments solicitations

Active vs passive sampling

• active: solicit respondents– Send out email, letters, phone

• Use sampling frame to develop a sample, I.e. list

– Keep track of who responds– Follow up on non-respondents if possible– Compare respondents/non-respondents

looking for biases

• PassivePopup box: “would you take a few minutes to

help us…”

Response Rates

• % of sample who actually participate• low rates may indicate bias in responses

– Whom did you miss? Why?– Who chose to cooperate? Why?

• How much is enough? – Babbie: 50% is adequate; 70% is very good

Increasing response rates

• Harder to say ‘no’ to a person• Captive audience• NOT an extra step • Explain purpose of study

– Don’t underestimate altruism• Why you need them• Incentives

– Reporting back to respondents as a way of getting response

– Money; entry in a sweepstakes

• Follow up (if you can)

Web survey problems

• Loss of context – what exactly are you asking about, what are they responding to?– Are you reaching them at the appropriate point in their

interaction with site?

• Incomplete responses • Multiple submissions

Passive: problems may include

• Response rate probably unmeasurable• May be difficult to compare respondents to

population as a whole• Likely to be biased (systematic error)

– Frequent users probably over-represented– Busy people probably under-represented– Disgruntled and/or happy users probably over-

represented

Questionnaire construction

Questionnaire construction

• Content– Goals of study: What do you need to

know?– What can respondents tell you?

• Conceptualization• Operationalization – e.g., how exactly

do you define “household with access to internet”?

• Question design• Question ordering

Topics addressed by surveys

• Respondent characteristics• Sampling element characteristics

– “Tell me about every member of this household…”

• Respondent/sampling element behavior• Respondent opinions, perceptions,

preferences, evaluations

Respondent characteristics

• Demographics: what do you need to know? How will you analyze data?– Age, sex, education, occupation, year in

school, race/ethnicity, type of employer…– Equal intervals

• User role (e.g., buyer, browser…)• Expertise – hard to ask

– Subject domain– Technology– System/site

Behavior

• Tasks (e.g., what did you do today?)• Site usage, activity

– Frequency; common functions – hard to answer accurately

– Self-reports vs observations• Web and internet use: Pew study

Opinions, preferences, concerns

• About the site: Content, organization, architecture, interface

• Ease of use• Perceived needs • Preferences• Concerns

– E.g., security

• Success, satisfaction– Subdivided by part of site, task, purpose…

• Other requirements• Suggestions