Characteristics of Studies that might Meet the What Works Clearinghouse Standards: Tips on What to...

Characteristics of Studies that might Meet the What

Works Clearinghouse Standards:

Tips on What to Look For

1

Two levels of evidence use WWC standards

2EvidenceFactors

STRONG THEORY

EVIDENCE OF PROMISE

MODERATE EVIDENCE OF EFFECTIVENESS

STRONG EVIDENCE OF EFFECTIVENESS

Number of Studiesn/a

(logic model only)

At least one At least one At least one At least one At least two

Study Findings on a Relevant Outcome

n/a

Statistically significant, substantively important (0.25

standard deviation or larger) positive association

Statistically significant positive impact with no

unfavorable and overriding impacts

Statistically significant positive

impact with no unfavorable and

overriding impacts

Statistically significant

positive impact with no


Statistically significant positive impact with no unfavorable and overriding impacts

What Works Clearinghouse Standards

n/a n/aMeets without reservations

(RCT)

Meets with reservations(RCT or QED)

Meets without reservations

(RCT)Meets with reservations

(RCT or QED)

Study Sample Size n/a n/a Large sample Large sample n/a

Number of Study Sites n/a n/a n/a Multi-site sample Multi-site sample n/a

Similarity of Population

n/a n/a Overlaps with proposed populations or settings

Overlaps with proposed populations or settings

Overlaps with proposed

populations and settings

n/a

Official definitions for each term are available in 34 CFR 77.1.(c)

The WWC and its review process

3

For each focus “topic,” the WWC searches for all studies, retrieves the actual reports or articles, and reads all of them

The WWC assesses the quality of the evidence from each study

The WWC summarizes findings succinctly

The What Works Clearinghouse (WWC) reviews evidence to help educators make research-informed choices

4

WWC reviews every study against written standards

Link to the Handbook.

5

WWC ratings of effectiveness studies

Meets WWC evidence standards without

reservations

Meets WWC evidence standards

with reservations

Does not meet WWC standards

Each relevant outcome in the study receives one of three ratings; the study as a whole receives the rating of the highest-rated outcome.

6

WWC ratings and study designs

Meets WWC evidence standards without

reservations

Meets WWC evidence

standards with reservations

Does not meet WWC standards

Randomized controlled trials (RCTs) with little sample attrition

Non-randomized studies with

comparison groups similar at baseline

Comparison groups are not similar at baseline

7

General characteristics of studies that meet WWC standards

8

“Studies” need to address effectiveness of an intervention, include technical detail

Publicly-available original research studies

Peer-reviewed or not

Full written technical document (report, article), final version of the results

Studies by developers of their own product or independent evaluators

INNewspaper articles

Literature reviews, briefs, summaries

Secondary research (re-analysis)

Implementation studies

Correlational studies (unless EOP)

Descriptive studies of baselines and trend lines

OUT

9

Characteristics that pertain to the way the study was designed and

carried out

10

Intervention groupCollege Freshmen

SAT/ACT Score, Pell

Fall 2015

Enhanced Supports

Persistence through Spring

2016

Comparison groupCollege Freshmen

SAT/ACT Score, Pell

Fall 2015

Enhanced Supports


2016

Study parameters needed to meet WWC standards

Distinct groups

Baseline equivalence

No confounding factors

Appropriate outcomes

11

Distinct Groups

At least two groups (one intervention and one comparison) Not a group compared to itself (same students as freshmen and sophomore) Not sequential/historical cohorts (freshmen in 2015 vs freshmen in 2014)

Clear definition of intervention and eligibility criteria – what services and who can get them?

Intervention groupCollege Freshmen

Comparison groupCollege FreshmenDistinct groups

12

Created randomly or Non-randomly Identify eligible sample Convenience sample Lottery or flip of coin Matching Prospective Retrospective

Baseline Equivalence

Necessary for QEDs to show because differences before intervention could carry through to outcomes, good practice (but not necessary) for RCTs to show

Must demonstrate groups were similar prior to intervention (at baseline)

WWC looks at the size of the difference Effect size (ES) units – see Handbook for calculation

SAT/ACT Score, Pell

Fall 2015

SAT/ACT Score, Pell

Fall 2015Baseline equivalence

0.00 ≤ ES Difference ≤ 0.05 0.05 < ES Difference ≤ 0.25 ES Difference > 0.25

Satisfies baseline equivalence Statistical adjustment required to satisfy baseline equivalence Does not satisfy baseline equivalence

13

No Confounding Factors

A component completely aligned with only one study condition Impossible to separate effect of intervention and confounding factor Cannot attribute impact solely to intervention Study Does Not Meet WWC Standards

Common confounds Single unit (school, classroom, teacher, mentor) in one or both conditions Intervention is bundled with other services not being studied Treatment and comparison occur at different points in time (e.g., historical

cohort)

Enhanced Supports No enhanced SupportsNo confounding factors

14

Appropriate Outcomes

Used to determine the impact of the intervention

Eligible outcomes (and data) have the following characteristics: Face validity – measures what it is supposed to measure Reliability – measures things consistently Not over aligned with the intervention Collected in a similar manner across groups


2016


2016Eligible outcomes

15

Beyond the WWC standards: other study features needed to meet Moderate or Strong Evidence of

Effectiveness

16

Reminder: WWC standards only one factor

17EvidenceFactors

STRONG THEORY

EVIDENCE OF PROMISE




(logic model only)



n/aStatistically significant,

substantively important (0.25 standard deviation or larger) positive

association





overriding impacts



overriding impacts



overriding impacts



(RCT)



(RCT)


Study Sample Size n/a n/a Large sample Large sample

Number of Study Sites n/a n/a Multi-site sample Multi-site sample

Similarity of Population n/a n/a Overlaps with proposed

populations or settings





Overlaps with proposed populations

and settings


Key factors: number of studies, findings

18EvidenceFactors

STRONG THEORY

EVIDENCE OF PROMISE




(logic model only)



n/aStatistically significant,

substantively important (0.25 standard deviation or larger)

positive association





overriding impacts



overriding impacts




(RCT)



(RCT)


Study Sample Size n/a n/a n/a/a Large sample Large sample Large sample

Number of Study Sites n/a n/a n/a Multi-site sample Multi-site sample Multi-site sample

Similarity of Population n/a n/a Overlaps with proposed







and settings


Key factors: study sample, sites, population

19EvidenceFactors

STRONG

THEORY

EVIDENCE OF PROMISE

MODERATE EVIDENCE OF

EFFECTIVENESS


Number of Studies n/a (logic model only) At least one At least one At least one At least one At least two

Study Findings on a Relevant Outcome n/a

Statistically significant, substantively important (0.25 standard deviation or

larger) positive association




overriding impacts



overriding impacts



overriding impacts



(RCT)



(RCT)


Study Sample Size n/a n/a Large sample Large sample Large sample

Number of Study Sites n/a n/a Multi-site sample Multi-site sample Multi-site sample

Similarity of Population n/a n/a


or settings


or settings






Where to learn more

• WWC Web site (http://ies.ed.gov/ncee/wwc/webinar.aspx), go to “Tools and References“ under the Resources tab to find:

► Webinars about strong studies► Review protocols► Standards Briefs – more on specific WWC standards

• FAQ on designing strong QEDs (http://www.dir-online.com/wp-content/uploads/2015/11/Designing-and-Conducting-Strong-Quasi-Experiments-in-Education-Version-2.pdf

20

THANK YOU!

21

Characteristics of Studies that might Meet the What Works Clearinghouse Standards: Tips on What to...

Documents

Transcript of Characteristics of Studies that might Meet the What Works Clearinghouse Standards: Tips on What to...