CCT 333: Imagining the Audience in a Wired World

Class 8: Quantitative Methods

Quantitative Methods

• Unlike qualitative, involves metrics• Everything reduced to numbers• Not just descriptive (e.g., counting) but

inferential (e.g., predictive, indication of correlation)

Experiments

• Controlled specific measurement of phenomenon

• Not just X related to Y (correlation) but X causes Y (causation) - provided overarching theoretical model is correct of course

• Examples of use in design?

Benefits and Limitations

• Controlled environment• Precise responses• Causality can be

inferred

• Must constrain outside influence

• Can be hard to measure and determine causation in complex multivariate phenomena

• Creates fake tasks in fake contexts - validity issues

Data Mining

• Observation without being there - quantitative artifacts - e.g., Web access logs, click regions, turnstile counts, etc.

• Records consequences of actual action• Examples of use?

Benefits and Limitations

• Results based in actual action vs. perception or intent

• Potential for complete sample – not easily achieved in other methods

• Once infrastructure set, data collection can be automatic

• Privacy concerns - data could be used for less than noble purposes

• Evidence of actions might not reflect motivations for why actions are done

Surveys/Questionnaires

• Common method of obtaining information from broad cross-section of people

• Quality of information directly depends on who is surveyed and the quality of questions asked

• Online tools help - e.g., http://www.surveymonkey.com

Sampling

• Who is included in sample?• How are they reached? • Responders vs. non-responders - are they

qualitatively different groups?• Sampling errors and confidence intervals (or,

what +/- 3.5%, 19 times out of 20 means…)• Examples of sampling error

Question Guidelines

• Understandable • Unambiguous• Data is relevant to research goals• Data can be easily analyzed• Limited in scope - take respondent’s attention

span and willingness to help into account!

Question Guidelines (2)

• Specific better than general• Open/closed-ended questions - benefits and

challenges• Opening and closing preamble and

instructions important - especially if you’re not there to supervise

• Test it before you use it!

Variable Data Types

• Nominal (categorical differences; student, staff, faculty, etc.)

• Ordinal (rank order implied; e.g., year of study)• Interval (ranks w/ equivalent intervals - e.g., grade

point averages)• Ratio (intervals amenable to mathematical

manipulations; e.g, age in years• Pick the right one for the question at hand!

Likert Scale Questions

• 1-5, 1-7, 1-9 scales• Midpoint - what does it mean? If no opinion,

give that option• Can force decisions with even number of

options (1-6)• Take care in too many consecutive items with

same polarity of options - leads to patterned responses

Scales• Set of related questions measuring attitudes, beliefs,

orientations etc.• Ex: multiple intelligences, personality tests (others?)• Scales must actually measure what they claim, not be

redundant• Scale reliability - consistency in response patterns• Verify authenticity (esp. in web searches) - many scales are

meaningless, published scales generally stronger

Data analysis

• Simple description (55% of respondents are male, 75% under 25 said yes, etc.)

• Cross-tabulation (males answered this, females answered that)

• Correlation (direction and strength of link between two variables - slope and r-values)

• Causation (inferential tests, confidence of variation)

Descriptive

(http://www.cnn.com/politics)

Cross Tabulation

http://www.cnn.com/ELECTION/2008/primaries/results/epolls/index.html#WIDEM

Correlation

http://www.burntorangereport.com/showDiary.do?diaryId=5106

Causation?

• Assuming causation can be dangerous in previous examples

• Hard to prove causation from correlation - must tie to logical explanation of correlation (which can itself be debatable)

• Ex: “mean-world” syndrome - does TV watching cause higher perceived dangers in the world, or do paranoid people seek shelter from the world in their houses and end up watching more TV?

Org. research example• Social network questionnaire - who trusted whom in six

domains• Correlated with three scales - interdependence,

independence and proactivity• Correlational study - what relations existed between scales

and position in network? • Causation - not easy to prove• Relations verified by respondent reflection and personal

experience• Organizational redesign implications

Implications for Final Project

• Experiments and data mining – probably not useful given circumstances, although latter might be used to gauge interest in topic

• Surveys – perhaps – depends on nature of population targeted

Next week

• Complex User Interaction issues and scenarios/requirements planning