Research Methods

ObservationsInterviews

Case StudiesSurveys

Quasi Experiments

Classes of Research Methods Experimental Non-Experimental

• Observations • Interviews• Case studies• Surveys• Quasi-Experiments

Intervention

Target system

Outcomes

Approach Comparison

Approach Research Question Control ofBehaviors

Experiments How? Why? Yes

Surveys Who? What? Where?How many? Howmuch?

No

Quasi-Experiments

How? Why? No

Case Studies How? Why? No

Adapted from Yin (1994)

Experiments

Experiments Most common method of evaluation Primary components

• Desired intervention or change • Inputs (what you vary or keep the same)• Outcomes (what you are looking for)

Vary one or more inputs and look for differences in the outcomes.

Randomize

ExperimentsStrengths Can be used to show

cause and effect Quantitative techniques

can be used to show strength of relationships

Accepted across a wide-range of disciplines

Limitations In the “real world” in

may be difficult to have random assignment

May not be able to create realistic conditions in a controlled setting

Observations

Observations Used to capture the ways in which people act and

interact within a particular environment/setting Observations may be either qualitative or

quantitative Researcher will either use detailed field notes, audio

or video recording to capture data Focus may be on general aspects of

behavior/phenomenon for qualitative; for quantitative, typically will focus on a particular aspect of behavior that can be quntified

Strengths Extremely flexible – researcher can shift

focus as new data come to light Good method to study behaviors and other

aspects of a system that may be difficult to quantify

Enables researcher to study the richness and complexity of human behavior

Data may be taken at multiple points in time

Limitations Behavior to be studied must be defined in a

precise, concrete manner to be recognizable Typically takes multiple researchers Researcher’s presence may impact

participants’ behaviors Requires meticulous attention to detail in

planning, data collection, and data analysis Results are limited to the system studied

ReliabilityHow reproducible is the data?

How consistent is the data? Data collection protocol must exist (but may

be modified as research progresses) Data analysis training to assure consistency

between researchers Data analysis reliability assessment if

multiple coders are used

Validity

How well does the data collected address the characteristics of interest?

Face validity Content validity Criterion validity Construct validity

Interviews

Interviews Can produce a great deal of useful

information Can be structured, open-ended, or semi-

structured Can be used for individuals or groups of

individuals (focus groups) IRB – confidentiality and informed

consent must be planned well in advance

Interview tips Make sure interviewees are representative Find a suitable location Take a few minutes to establish rapport Don’t put words in people’s mouths Record responses verbatim Keep your reactions to yourself Use contingency questions to avoid irrelevant

questions

Choosing Interviewees Interviewees must be competent to

answer Interviewees must be willing to answer Questions should be relevant to the

Interviewees

Strengths Rich and complete information Question complexity can be greater than

survey Flexible (particularly if open-ended)

Limitations Can produce information that is not

related to research topic and/or not comparable to other data collected

More expensive than written surveys Time consuming Sensitive questions may not be answered

honestly

Case studies

Case Studies An evaluation method that

• “investigates a contemporary phenomenon within its real-life context,

• especially when the boundaries between phenomenon and context are not clearly evident”

(Yin 1994, p. 13)

Case studies Used to study an individual, program,

event, or organization in depth for a pre-defined period of time

Researcher must know what he/she is looking for

Use multiple sources of data, e.g. interviews, direct observation, document analysis, or surveys

Case StudiesStrengths Can be done after the

fact Valuable technique to

highlight learnings from a project

Data collection methods are relatively straight-forward

Limitations Difficult to generalize

the results to other parts of the organization or to other organizations.

Will not show cause and effect relationships

Time consuming

Quality Digest Case Study Example

Taking SPC to the production line Target System: Organizations that have

front line workers using SPC tools Types of data: Direct observation,

interviews, and documents Findings: Lessons learned, road blocks,

and best practices were highlighted

Surveys

Why use surveys? A method for collecting data to describe

some specific characteristics of individuals or groups of people.

A method for measuring some specific attitudes, perceptions, and beliefs of individuals or groups of people.

Survey design issues Sample selection Sample size Data collection method Questionnaire format Question construction

Surveys Understand how people view a particular

topic without asking every person Good method to quantify otherwise

qualitative information Data may be taken at multiple points in time Use questionnaires or structured interviews

to collect information

Adapted from Creswell (1994)

New Program Evaluation Example

SWE e-mail mentoring program Target System: Program participants Intervention: Mentors/mentees exchange e-mails

about careers in engineering Outcomes:

• Are mentees more likely to consider engineering careers than before?

• Do mentors and mentees feel more positive about themselves?

SurveysStrengths Can be used to create

quantitative measures of “softer” types of data.

Can be done at many points in time if needed

Methods for evaluating the data exist

Limitations Results are limited to

the overall group represented in the survey sample

Does not give you a way to follow up if results are inconclusive.

Quasi-Experiments

Quasi-Experiments Used in social science, education, policy, and

management systems Use when randomization of target system and or

intervention is not possible Multiple-designs exist Primary components

• Study group• Comparison group• Intervention

Quasi-Experimental Approach

Nonequivalent Control Group Example

O1 X O2

O1 O2

Autonomous Team ExampleImplementing teams in a mineral plant Study group: Autonomous teams in a startup

plant Comparison group: Existing shift workers in

the original plant Intervention: Creation of teams Outcomes: Job satisfaction, trust, and

productivity

“Ideal Outcomes”

1.5

2

2.5

3

3.5

4

4.5

T1 T2

Survey at Time T(x)

Scor

e

Study SiteComparison Site

Trends in Opposite Direction

1.5

2

2.5

3

3.5

4

4.5

T1 T2

Survey at Time T(x)

Scor

e

Study SiteComparison Site

Trends with Differing Growth Rates

1.5

2

2.5

3

3.5

4

4.5

T1 T2

Survey at Time T(x)

Scor

e

Irving ParkPalatine

Quasi-ExperimentsStrengths Can be used in the

“real world” Does not require

changes to existing organization

Designs and techniques for evaluation exist

Limitations Can not be used to

show cause and effect May not be able to

generalize the results Results can potentially

be explained by other factors