7/28/2019 11-FixedDesigns.doc

1/4

INLS 98, Fall 2004

11, FIXED RESEARCH DESIGNS

Based on a theory about a relationship among variables

Variable: some defined property or characteristic of a person, thing, group or situationthat can be measured in some way, and for which these measurements vary, so that

they can be compared to one another (Robson, 2002, p.100)

Theories specify which variables are important and the ways in which they should be

related to each other

Working from a positivist perspective

Stated most strongly by Kerlinger (1986), p.293: "The controlled experiment is the

desired model of science."

The various possible designs are evaluated in terms of threats to internal and external

validityThreats to internal validity may affect scores on the observation

Threats to external validity may affect our ability to generalize the results

Internal validity: the approximate truth about inferences regarding cause-effect or causal

relationships (Trochim, 2001, p.348)

ThreatsHistory: events that take place in the course of the study that

might affect the outcomeTesting: effects of repeated testing of the same variable

(practice, sensitization)Instrumentation: measuring the same variable with two different

instruments, so that variation can be attributed to difference ininstruments

Regression to the mean: a statistical artifact; extreme scorestend to regress to the mean on re-assessment

Mortality: attrition of subjects from original random sampleMaturation: changes in subjects over time

Selection: non-random selection and/or non-random assignmentto treatmentsSelection by maturation interaction: different subgroups may

mature at different ratesAmbiguity about causal direction: A may cause B or B may cause

A

7/28/2019 11-FixedDesigns.doc

2/4

INLS 98, Fall 2004

Research Designs, page 2

Diffusion of knowledge about treatments: subjects' expectationsof treatment may affect their responses

Compensatory equalization of treatments: researchers maycompensate and give the control group some type of specialtreatment not included in the study design

Compensatory rivalry: participants in the control group may workto improve their performance

External validity: the degree to which the conclusions in your study would hold for other personsin other places and at other times (Trochim, 2001, p.347)

ThreatsSelection: the findings are relevant only to the study participants

Setting: the findings are relevant only to the situation studied

History: the findings are relevant only to those having exactly the same history

Construct effects: the particular constructs studied are specific only to the study

participantsReactive effects of experimental arrangements: artificiality of lab studies

Multiple-treatment interference: if multiple treatments are used, the first couldinfluence reactions to the second

EXPERIMENTAL DESIGNS

Posttest-only control group design (C&S 6)

Randomization is critical

Pretest-posttest control group design (C&S 4)

Little threat to internal validity

Still need to watch for intra-session history, interpretation of regression to themean

Randomization is necessary

Still some possible threats to external validity

Interaction of testing and treatment

Interaction of selection and treatmentReactive arrangements: experimental setting is artificial

Factorial designs

Multiple independent variables

An elaboration of the previous two

7/28/2019 11-FixedDesigns.doc

3/4

INLS 98, Fall 2004

Research Designs, page 3

Designs involving repeated measures; counter-balanced designs (C&S 11)

With random assignment, can be considered a true experimental design

Posttest only

Multiple treatments; order of implementation is counterbalanced, to avoid order effects

Strong on internal validitySome possibility of interactions with selection (if no random assignment)

Also a possibility of interactions with treatment, threatening external validity

QUASI-EXPERIMENTAL DESIGNS

Nonequivalent control group design (C&S 10)

Much improved over simple pretest-posttest, even without randomization

Possible regression to the mean, since no random assignment

Possible selection by maturation interaction could affect external validity

Time-series experiment (C&S 7)

One major issue associated with internal validity: are there other plausible explanations

for observed changes in the trend?

PRE-EXPERIMENTAL DESIGNS

Single-group posttest only; The one-shot case study (C&S 1)

Essentially no control over anything

Don't use this design

Posttest-only non-equivalent groups; Static-group comparison (C&S 3)

No random assignment to groups, so no guarantee that the treatment caused the change

Weaknesses in the design

Non-random assignment

Experimental mortality, attrition

7/28/2019 11-FixedDesigns.doc

4/4

INLS 98, Fall 2004

Research Designs, page 4

The one-group pretest-posttest design (C&S 2)

Weak design, because several plausible explanations cannot be ruled out

History

Maturation

TestingInstrumentation, instrument decay

Statistical regression - regression to the mean

RESEARCH PROTOCOL ISSUES

Individual sessions or group sessions?

What will the participant do during the session?

Questionnaires

Interactions with a system

Other types of data collection

Will any incentive be offered?

STRUCTURE OF THE METHODS PROPOSAL

Outlines available from assignments Web pageFor an experimental study

References:

Campbell, D. T., & Stanley, J. C. (1963).Experimental and Quasi-Experimental Designs for

Research. Dallas: Houghton Mifflin.Kerlinger, F. N. (1986).Foundations of Behavioral Research. 3rd ed. New York: Holt, Rinehart

and Winston.