1

Implementation of Randomized Trials

David Myers

American Institutes for Research

Washington, DC

Prepared for IES/NCER Summer Research Training Institute, 2008

2

Context

• Conducting effectiveness trials and not efficacy trials• Design and implementation of random assignment

should not distort the program/intervention or the population served

• Programs should run at capacity

3

Illustrative Studies: Personal Involvement

• Upward Bound– Nationally representative study– 67 sites and more than 3,000 students– Student RA within site– http://www.mathematica-mpr.com/publications/PDFs/upboundimpact.pdf

• Workplace Literacy– 3 sites and about 500 adult learners– Learner RA within site– "Addressing Literacy Needs at Work: Implementation and Impact of Workplace

Literacy Programs. Final Report." Washington, DC: Mathematica Policy Research, Inc., October 1998

4

Illustrative Studies: Personal Involvement

• 21st Century Community Learning Centers – 12 grantees (elementary school study)– Student RA within site– http://www.mathematica-mpr.com/publications/PDFs/elementaryschools.pdf

• NYC Voucher Experiment– 2000 students– Family RA– http://www.mathematica-mpr.com/publications/PDFs/nycfull.pdf

5

Illustrative Studies: Personal Involvement

• Reading Comprehension– 10 districts and 40 schools– School RA– http://www.mathematica-mpr.com/publications/PDFs/readcomp.pdf

• Closing the Reading Gap– 32 schools (4 interventions)– Student RA to reading groups within schools– http://ies.ed.gov/ncee/pdf/20084013.pdf

6

Illustrative Studies: Personal Involvement

• Quantum Opportunities– 7 sites and about 1000 students– Student RA within site– http://www.mathematica-mpr.com/publications/PDFs/QOPfinalimpacts.pdf

7

Practical Problems in Implementation

• Technical• Political and Ethical Challenges• Recruitment (not independent of P and E Challenges)

8

Further Exploration: Technical

• Cross-overs– Members of C get into T or something like T (use UB

example and James Comer example)• Into T -- straight forward adjustment (AIR, 1996)• Something like T -- ?; affects interpretation

• Attrition– Post baseline -- “straight forward” adjustment (e.g., MI)– Before baseline -- problematic, no information– How to fix?????

9

Further Exploration: Technical

• Unbalanced designs– Don’t demand additional recruitment

• Large variance in selection probabilities (weights)– “over subscription” didn’t meet expectations (UB example)

• Dishonest assignment -- post randomization– Sites don’t tell all students they have been selected for the

program (UB example)– Sites ignore RA and move controls into T

10

Further Exploration: Ethical and Political Challenges

• Random assignment isn’t fair– Programs afraid of denying services to students

• Is it fair to never give a student a chance? (UB example)

• Random assignment will force a program/teacher to serve a different population– Role of stratification to serve the desired mix

• Some students (units) must be served– Role of the “wild card” before randomization– Wild cards excluded from analysis

11

Further Exploration: Ethical and Political Challenges

• All “seats” must be filled– Role of the waiting list

• Random selection within strata, if needed and desired by program operators

• T and C groups analyzed as implemented at initial randomization

• When programs believe recruitment will distort the population– Identify “most likely” and “least likely” to serve– Prior to randomization and stratify in analysis

12

Further Exploration: Ethical and Political Challenges

• New treatments/interventions are hard to sell– Core programs vs. supplemental programs– Does the core program align with other curriculum and state

assessments?• Concerns about making AYP

– Will a supplemental program reduce hours of instruction in a core area such as ELA?

13

Strategies for Recruitment: Schools

• Start with assistant superintendent for instruction (reading, math, science) or someone of similar stature

• Quickly develop a relationship with an office and not just an individual -- staff come and go

• Determine who needs to approve and to buy into participation– Superintendent -- they may need to go to the school board– Principals– Teachers– Parents and community groups

14

Strategies for Recruitment: Schools

• Be prepared to meet with them– Have a recruitment team

• Technical expertise in the design– Be prepared to tell audience why RA is valuable and

not in a technical sense!• Deep knowledge about the intervention -- most important

• Example of a meeting (next slide)

15

Irrefutable Evidence

16

Strategies for Recruitment: Schools

• Establish expectations for the researchers and the schools– Minimize legal talk

• Will bring in their general counsel and … .

– Responsibilities and timelines• Obtaining consent

• Data collection

• Administering instruments and tests

• Answering questions from parents and others