Quasi-experimental Design

CRJS 4466EA

Introduction

Quasi-experimentDescribes non-randomly assigned

participants and controls subject to impact assessment

Most common design involves constructed comparisons

Matching participants and comparisons Statistical adjustment

Rationale for quasi-experimental usage

Random assignment not within the evaluator’s capability

Powerful stakeholders oppose randomization

Issues

Quasi-experiment often results from a randomized experimentProgram staff subvert the randomization

process (assigning only those random subjects that will yield good results)

Attrition from treatment Problems with data collection

Measuring impacts in quasi-experiments

Net effect = gross outcome for an intervention group – gross outcome for a constructed control group + or – uncontrolled difference between intervention and control groups + or – design effects and stochastic error

“when there is a possibility that one or more relevant differences exists between the members of the intervention and comparison groups, as there typically is in quasi-experiments, then it is also a possibility that these differences – not the intervention – cause all or part of the observed effects” (Rossi, Freeman and Lipsey, 1999)

“In evaluations in which selection bias is at work, net effects would tend to be over-estimated because a portion of the difference between the intervention group and its comparison would result from the stronger potential for positive (or, sometimes, negative) effects inherent in the persons selected for intervention” (Rossi, Freeman and Lipsey, 1999)

Ex ante quasi-experiments

Occurs before intervention to plan selection of the comparisonsGenerally maximizes potential for equivalenceConsiders motivations Identifies characteristicsLocates potential comparisonsAllows review of prior, like evaluations

Ex post quasi-experiments

Decision to undertake evaluation occurs after program is underwayTargets are enrolled Insufficient time to enroll a fresh group and

follow them to termination Issues can be managed to some extent

through statistical controls

Constructing control groups by matching

Participants are sought first and comparisons are matched afterwards

Matching is based on prior knowledge and theoretical understanding of the social processes in question

Matching information is often sought in the published literature

Attend to variables that are potentially related to self-selection processes

Use only as many variables for matching as are necessary Pertinent characteristics will tend to be intercorrelated and,

therefore, somewhat redundant

Characteristics useful in devising constructed control groups –

Exhibit 9A of textCharacteristics of individuals Age, sex, educational attainment, socio-economic status, ethnicity,

etc.

Characteristics of families (households) Life-cycle stage, number of members, number of children, etc.

Characteristics of organized units (schools, classes, unions, etc.) Size differentiation, levels of authority, growth rate, budget, etc.

Characteristics of communities (territorially organized units) Population size, territorial size, industry mix, governmental

organization, etc. Note: not a substitute for priori knowledge directly relevant to

the phenomena being studied

Matching procedures

Options are either individual or aggregate matchingIndividual matching – draws a “partner” for each participant from the unexposed poolAggregate matching – overall distributions in the participant and control groups on each matching variable are made to correspondIndividual matching is usually preferable (the more characteristics especially) but is more expensive, time consuming and difficult to execute for a large number of matched variables (Rossi, Freeman and Lipsey, 1999)

Equating groups by statistical procedures

Statistical procedures, rather than matching, are now generally used in both ex ante and ex post quasi-experimental evaluations as the primary approach to dealing with selection bias and other unwanted differences between groups (Rossi, Freeman and Lipsey, 1999)

Multivariate statistical methods are commonly used to adjust for a number of contaminating variables simultaneously

Matched and statistical controls are equivalent ways of proceeding, with statistical controls possessing some superior qualities arising from the retention of observations that might have to be discarded under matching procedures (Rossi, Freeman and Lipsey, 1999)

Multivariate statistical models

Allows for creation of a statistical model to account for initial measurement differences between the intervention and comparison groups

The model adjusts the outcome difference between those groups to subtract the portion attributable entirely to those initial differences

Whatever difference on outcomes remains after this subtraction, if any, is interpreted as the net effect of the intervention (Rossi, Freeman and Lipsey, 1999)

Need also to control for variables dealing with selection of individuals into the intervention vs. the control groupExamples of these control variables include: proximity of individuals to the program site, motivation to enroll in the program, whether they had the characteristics program personnel used to select participants, etc.Of course, the variables related to selection are only useful if they also relate to outcome (hamburger/hotdog example) (Rossi, Freeman and Lipsey, 1999)

Regression-discontinuity designs

Evaluator is given selection variables up-frontLimited applications, but most rigourous methodAlso called cutting points designs“regression-discontinuity designs approximate randomized experiments to the extent that the known selection process is modeled properly, which is generally straightforward because it is explicit and quantitative” (Rossi, Freeman and Lipsey, 1999)

Generic controls

Examples: age, sex, income, occupation and race; distributions of certain characteristics and processes (e.g. birth rates, sex ratios, proportions of persons in various labour force categories; and, derivatives of these measuresBest examples of appropriate use of generic controls are from epidemiological studiesE.g. detection of epidemics rests heavily on the epidemiologist’s knowledge of ordinary incidence rates for various diseases

Generic controls used successfully by epidemiologists because selection processes are either known or unimportant (i.e., use of morbidity rates to detect epidemics)

Issue of insufficient norms for using generic controls (i.e. achievement tests for inner-city children)

Generic controls are tempting because of low cost and limited time to collect data

A final note

Numerous comparisons between randomized designs and quasi-experimental designs relative to net effects measuring the same thing

“Lipsey and Wilson (1993) compared the mean effect size estimates reported for randomized versus non-randomized designs in 74 meta-analyses of psychological, educational, and behavioural interventions. In many of the sets of studies included in a meta-analysis, the effect estimates from nonrandomized comparisons were very similar to those from randomized ones. However, in an equal number of cases there were substantial differences in both directions” (Rossi, Freeman and Lipsey, 1999)