Slides to accompany Weathington, Cunningham & Pittenger (2010),

Chapter 8: Creating and Using Assessments, Surveys, and Objective Measures

1

Objectives

• Purpose of Measurement

• Creating a Measurement Scale

• Constructing Interviews, Questionnaires, and Attitude Surveys

• Question Response Formats

• Writing Good Questionnaire and Survey Items

• Determining the Sample Size for a Survey

• Naturalistic Observation2

Purpose of Measurement

• To work with our operational definitions

• To facilitate consistency in research

• Issues to overcome:

– Mistrust in measurement

– Excessive trust in measurement

•Reification

•Missing important information

3

Creating Measures

Follow the sequence:

1. What questions are you asking?

• Hypotheses

2. How can you collect the best data?

• Operational definition + method plan

3. What measurement approach would be most accurate?

4

Interviews

• Several varieties (Table 8.1)

• Pros:

– Encourage participation, richer answers

– Ability to clarify questions

• Cons:

– Expensive (time, $)

– Training required

– Interpersonal issues may arise5

Self-Report Surveys

• Paper-pencil, phone- or internet-based

• Pros:

– Cheap

– Easy distribution

• Cons:

– Data quality may suffer

– Lack of control over data collection6

Survey Strategies• Use a sufficiently large sample

• Can increase retention by:

– Using a “captive” audience

– Making multiple contacts and reminders

– Using creative labeling/packaging

– Offering incentives/gifts

– Facilitating quick completion (good timing, easy format)

7

Question Types

• Open-ended

– Can gather rich data

– Responses often incomplete or difficult to interpret

– “What symptoms or signs do your recognize in yourself when you are experiencing a great deal of stress?”

8

Question Types

• Closed-ended/response

– Researcher supplies response options:

•Nominal categories

•Forced choice

•Likert

•Guttman9

Question Writing Strategies

• Use an existing measure

• Single questions/statements

• Be specific and clear

– K.I.S.S.

• Write with neutrality (avoid bias)

• Don’t embarrass/anger the participant

10

Question Writing Strategies

• Make it easy to answer

• Ask more than one question

• Try to avoid a response set

• Avoid full transparency (obviously correct)

11

Determining Survey Sample Size• Using formulas, you can estimate the optimal

sample size for surveys

• Formulas differ depending on the type of responses you are gathering (yes/no, Likert, open-ended responses)

• Your textbook presents formulas for estimating N for surveys with binary outcomes (e.g., “Yes/No” type questions)

12

Naturalistic Observation

• A.K.A. Field studies, observation, natural experiment

• Follow same strategies as with survey construction:

– What behaviors to observe?

– How defining these behaviors?

– What data would be best to gather?

13

Participant Observation

• Joining the group to learn about its functioning or about phenomena in that environment

• Ethnographic approach

• Can be difficult to stay objective

• Can be risky

• May lead to criticism

14

Observational Data

• Frequency

– Counting repetition in specified time span

• Duration

– Length of time behavior lasts

• Interval

– How long between behaviors

• Intensity

– How strong was the behavior or stimulus

15

Observational Research Issues

• Interrater reliability

– Index of consistency across multiple raters

•Cohen’s kappa for level of agreement (nominal/ordinal scales)

• r for duration or interval (interval and ratio scales)

– Improved by using “blind” raters, training, video recording, avoiding reactivity

16

What is Next?

• **instructor to provide details

17