Quasi-Experimental Designs Manipulated Treatment Variable but Groups Not Equated.

Quasi-Experimental Designs

Manipulated Treatment Variablebut

Groups Not Equated

Pretest-Posttest Nonequivalent Groups Design

N O X ON O O

• Cannot assume that the populations are equivalent prior to treatment.

• Selection and Selection Interactions are threats to internal validity.

• Can try to select subjects or intact groups in ways that make it likely that the groups are similar, but what about unknown variables on which the groups may differ.

Double-Pretest Nonequivalent Groups DesignN O O X ON O O O

• Some control for Selection x Maturation.• If groups are maturing at different rates,

that may be shown in the two pretests.

Regression-Discontinuity Design

C O X OC O O

• ‘C’ indicates subjects are assigned to groups based on score on covariate.

• Groups are deliberately nonequivalent.• I shall illustrate with a hypothetical

example

Evaluating an Online Tutorial

• IV = Student completes online tutorial or not.

• DV = Student’s score on statistics course exam.

• Pretest/Covariate = Student’s score on a test of statistics aptitude.

• How do I assign students to groups?


Let the students self-select into groups.• This gives me a pretest-posttest

nonequivalent groups design.• I use pretest scores as covariate in

ANCOV.• This does not, however, remove all

possible confounds.• How might the groups have differed other

than on statistics aptitude???


Randomly assign students to groups.• This would be an experimentally sound,

randomized pretest-posttest control group design.

• And you would live to regret trying it.– Students complain.– Their parents complain.– The Chair of the Department intervenes.– The IRB revokes your authority to do

research.


Try a switching-replications design.• Those who have to wait until the second

half of the class would be disadvantaged– if you don’t learn the beginning material well,

the later material will very hard to learn.• Those who have it taken away at mid-

semester will complain.


Apply the treatment only to those most in need of it, those lowest in statistics aptitude.

• Those not selected may complain that they could benefit from it too.– Tough, there is an American tradition of

favoring the underdog.


• Those selected may complain about having to do extra work.– You can’t please everybody every time.

Convince them they need to do it.• May be cases when you want to give the

tutorial only to those with highest aptitude– purpose of tutorial is to allow brightest

students to finish course early, allowing prof more time to spend in class with others.


• Suppose I pick a cutoff on the covariate so that ½ get the treatment, ½ don’t.

• Simulated data are in file RegD0.txt .• C = control group, T = Treatment group.• 2nd score is posttest score.• 3rd score is pretest score.• I defined the treatment effect to be zero in

the population.

http://core.ecu.edu/psyc/wuenschk/docs2210/Data2210/RegD0.txt


• In the population, Post = 7 + 1.35Pre + error, = .9,

• In the sample, ignoring group, Post = 7.58 + 1.27Pre + error, r = .85, and MSE = 2.13

• Look at this plot of the data:

71.12 error


• Now I compute two separate regressions, one for each group.

• T: Post = 8.09 + 1.17Pre + error, r = .62, and MSE = 2.13.

• C: Post = 6.33 + 1.43Pre + error, r = .72, and MSE = 2.29.

• The plot shows how little the two lines differ:


• I re-simulated the data, with a 3 point treatment effect built in.

• The data are at RegD1.txt.• T: Post = 11.27 + 1.07Pre + error, r = .82,

and MSE = 1.35.• C: Post = 7.90 + 1.18Pre + error, r = .82,

and MSE = 1.25 • The plot shows a clear regression

discontinuity:

http://core.ecu.edu/psyc/wuenschk/docs2210/Data2210/RegD1.txt


• The dotted line shows the expected regression for the treatment group if there were no treatment effect.

• Hard to imagine how any threat to internal validity would create the observed regression discontinuity.

• Caution: This analysis assumes the regression is linear, not curvilinear.

Proxy-Pretest Design

N O1 X O2

N O1 O2

• You have a nonequivalent groups posttest only control group design.

• The treatment has already been administered.

• Now you decide you want a pretest too.• Can’t warp time, can find an archival proxy

pretest.

PSYC 2210 and Understanding Stats

• Mid-semester, I ask myself “does taking 2210 improve students understanding of stats?”

• I’ll compare students in current 2210 class with those in another class (excluding any who have already taken 2210).

• I want a pretest too, but the treatment is already in progress.

PSYC 2210 and Understanding Stats

• I use, as a proxy pretest, students’ final averages from PSYC 2101.

• Conduct an ANCOV– IV = took 2210 or not– DV = end of course stats achievement test– COV = the proxy pretest

Separate Pre-Post Samples Design

• Pretest subjects different than posttest subjects.

• I want to evaluate online tutorial in stats.

• Both I and my friend Alex taught stats this semester and last semester.

• Both of us gave our students a end of course standardized exam.

N ON X O

N ON O

Separate Pre-Post Samples Design

• Row 1: My students last semester, no tutorial.

• Row 2: My students this semester, with tutorial.

• Row 3: Alex’s students last semester, no tutorial

• Row 4: Alex’s students this semester, no tutorial.

• Selection problems likely.

N ON X O

N ON O

Nonequivalent Groups Switching Replications DesignN O X O ON O O X O

• I am teaching two sections of stats.• I make the experimental tutorial available

the first half semester to one class• and the second half semester to the other.• Might reduce complaints, until students

from the two classes meet each other.

Nonequivalent Dependent Variables Design

O1 X O1 N O2 O2

• Only one group of subjects, but two DVs.• One DV you expect to be affected by X.• The other you expect not to be affected by X.• The second DV serves as a control variable.• Should be similar enough to 1st DV that it will be

affected in same way by history, maturation, etc.


• I want to evaluate effect of stats remedial tutorial given to all PSYC 2210 students.

• DV1 = stats knowledge measured at start and end of semester.

• DV2 = Vocabulary test given at start and end of semester.

• More impressive if have multiple control variables and an a priori prediction of extent to which each will be affected.


• Stats Knowledge (DV1) – most affected

• Logical Thinking – next most• Verbal Reasoning – same as LT• Arithmetic Skills – next most• Vocabulary – next most• Artistic Expression – least affected by

treatment

Regression Point Displacement Design

N(n = 1) O X O

N O O

• Only one subject in the treatment group• Several or many in the control group.• X = novel economic development plan.• Treatment unit = your hometown, in which

the plan was just initiated.


• You consult state economic database.• Pick 20 cities comparable to your city,

these serve as the control group.• Pretest = value of criterion variable (such

as unemployment rate) last year.• Posttest = value of same variable two

years later.


• Plot Post x Pre for the Control Group.• Draw in regression for predicting Post from

Pre.• Plot experimental unit data point.• If it is displaced well away from regression

line, you have evidence of a treatment effect.

Quasi-Experimental Designs Manipulated Treatment Variable but Groups Not Equated.

Documents

Transcript of Quasi-Experimental Designs Manipulated Treatment Variable but Groups Not Equated.