Post on 18-Jan-2018
description
Characteristics of Studies that might Meet the What
Works Clearinghouse Standards:
Tips on What to Look For
1
Two levels of evidence use WWC standards
2EvidenceFactors
STRONG THEORY
EVIDENCE OF PROMISE
MODERATE EVIDENCE OF EFFECTIVENESS
STRONG EVIDENCE OF EFFECTIVENESS
Number of Studiesn/a
(logic model only)
At least one At least one At least one At least one At least two
Study Findings on a Relevant Outcome
n/a
Statistically significant, substantively important (0.25
standard deviation or larger) positive association
Statistically significant positive impact with no
unfavorable and overriding impacts
Statistically significant positive
impact with no unfavorable and
overriding impacts
Statistically significant
positive impact with no
unfavorable and overriding impacts
Statistically significant positive impact with no unfavorable and overriding impacts
What Works Clearinghouse Standards
n/a n/aMeets without reservations
(RCT)
Meets with reservations(RCT or QED)
Meets without reservations
(RCT)Meets with reservations
(RCT or QED)
Study Sample Size n/a n/a Large sample Large sample n/a
Number of Study Sites n/a n/a n/a Multi-site sample Multi-site sample n/a
Similarity of Population
n/a n/a Overlaps with proposed populations or settings
Overlaps with proposed populations or settings
Overlaps with proposed
populations and settings
n/a
Official definitions for each term are available in 34 CFR 77.1.(c)
The WWC and its review process
3
For each focus “topic,” the WWC searches for all studies, retrieves the actual reports or articles, and reads all of them
The WWC assesses the quality of the evidence from each study
The WWC summarizes findings succinctly
The What Works Clearinghouse (WWC) reviews evidence to help educators make research-informed choices
4
WWC reviews every study against written standards
Link to the Handbook.
5
WWC ratings of effectiveness studies
Meets WWC evidence standards without
reservations
Meets WWC evidence standards
with reservations
Does not meet WWC standards
Each relevant outcome in the study receives one of three ratings; the study as a whole receives the rating of the highest-rated outcome.
6
WWC ratings and study designs
Meets WWC evidence standards without
reservations
Meets WWC evidence
standards with reservations
Does not meet WWC standards
Randomized controlled trials (RCTs) with little sample attrition
Non-randomized studies with
comparison groups similar at baseline
Comparison groups are not similar at baseline
7
General characteristics of studies that meet WWC standards
8
“Studies” need to address effectiveness of an intervention, include technical detail
Publicly-available original research studies
Peer-reviewed or not
Full written technical document (report, article), final version of the results
Studies by developers of their own product or independent evaluators
INNewspaper articles
Literature reviews, briefs, summaries
Secondary research (re-analysis)
Implementation studies
Correlational studies (unless EOP)
Descriptive studies of baselines and trend lines
OUT
9
Characteristics that pertain to the way the study was designed and
carried out
10
Intervention groupCollege Freshmen
SAT/ACT Score, Pell
Fall 2015
Enhanced Supports
Persistence through Spring
2016
Comparison groupCollege Freshmen
SAT/ACT Score, Pell
Fall 2015
Enhanced Supports
Persistence through Spring
2016
Study parameters needed to meet WWC standards
Distinct groups
Baseline equivalence
No confounding factors
Appropriate outcomes
11
Distinct Groups
At least two groups (one intervention and one comparison) Not a group compared to itself (same students as freshmen and sophomore) Not sequential/historical cohorts (freshmen in 2015 vs freshmen in 2014)
Clear definition of intervention and eligibility criteria – what services and who can get them?
Intervention groupCollege Freshmen
Comparison groupCollege FreshmenDistinct groups
12
Created randomly or Non-randomly Identify eligible sample Convenience sample Lottery or flip of coin Matching Prospective Retrospective
Baseline Equivalence
Necessary for QEDs to show because differences before intervention could carry through to outcomes, good practice (but not necessary) for RCTs to show
Must demonstrate groups were similar prior to intervention (at baseline)
WWC looks at the size of the difference Effect size (ES) units – see Handbook for calculation
SAT/ACT Score, Pell
Fall 2015
SAT/ACT Score, Pell
Fall 2015Baseline equivalence
0.00 ≤ ES Difference ≤ 0.05 0.05 < ES Difference ≤ 0.25 ES Difference > 0.25
Satisfies baseline equivalence Statistical adjustment required to satisfy baseline equivalence Does not satisfy baseline equivalence
13
No Confounding Factors
A component completely aligned with only one study condition Impossible to separate effect of intervention and confounding factor Cannot attribute impact solely to intervention Study Does Not Meet WWC Standards
Common confounds Single unit (school, classroom, teacher, mentor) in one or both conditions Intervention is bundled with other services not being studied Treatment and comparison occur at different points in time (e.g., historical
cohort)
Enhanced Supports No enhanced SupportsNo confounding factors
14
Appropriate Outcomes
Used to determine the impact of the intervention
Eligible outcomes (and data) have the following characteristics: Face validity – measures what it is supposed to measure Reliability – measures things consistently Not over aligned with the intervention Collected in a similar manner across groups
Persistence through Spring
2016
Persistence through Spring
2016Eligible outcomes
15
Beyond the WWC standards: other study features needed to meet Moderate or Strong Evidence of
Effectiveness
16
Reminder: WWC standards only one factor
17EvidenceFactors
STRONG THEORY
EVIDENCE OF PROMISE
MODERATE EVIDENCE OF EFFECTIVENESS
STRONG EVIDENCE OF EFFECTIVENESS
Number of Studiesn/a
(logic model only)
At least one At least one At least one At least one At least two
Study Findings on a Relevant Outcome
n/aStatistically significant,
substantively important (0.25 standard deviation or larger) positive
association
Statistically significant positive impact with no
unfavorable and overriding impacts
Statistically significant positive
impact with no unfavorable and
overriding impacts
Statistically significant positive
impact with no unfavorable and
overriding impacts
Statistically significant positive
impact with no unfavorable and
overriding impacts
What Works Clearinghouse Standards
n/a n/aMeets without reservations
(RCT)
Meets with reservations(RCT or QED)
Meets without reservations
(RCT)
Meets with reservations(RCT or QED)
Study Sample Size n/a n/a Large sample Large sample
Number of Study Sites n/a n/a Multi-site sample Multi-site sample
Similarity of Population n/a n/a Overlaps with proposed
populations or settings
Overlaps with proposed
populations or settings
Overlaps with proposed
populations and settings
Overlaps with proposed populations
and settings
Official definitions for each term are available in 34 CFR 77.1.(c)
Key factors: number of studies, findings
18EvidenceFactors
STRONG THEORY
EVIDENCE OF PROMISE
MODERATE EVIDENCE OF EFFECTIVENESS
STRONG EVIDENCE OF EFFECTIVENESS
Number of Studiesn/a
(logic model only)
At least one At least one At least one At least one At least two
Study Findings on a Relevant Outcome
n/aStatistically significant,
substantively important (0.25 standard deviation or larger)
positive association
Statistically significant positive impact with no
unfavorable and overriding impacts
Statistically significant positive
impact with no unfavorable and
overriding impacts
Statistically significant positive
impact with no unfavorable and
overriding impacts
Statistically significant positive impact with no unfavorable and overriding impacts
What Works Clearinghouse Standards
n/a n/aMeets without reservations
(RCT)
Meets with reservations(RCT or QED)
Meets without reservations
(RCT)
Meets with reservations(RCT or QED)
Study Sample Size n/a n/a n/a/a Large sample Large sample Large sample
Number of Study Sites n/a n/a n/a Multi-site sample Multi-site sample Multi-site sample
Similarity of Population n/a n/a Overlaps with proposed
populations or settings
Overlaps with proposed
populations or settings
Overlaps with proposed
populations and settings
Overlaps with proposed populations
and settings
Official definitions for each term are available in 34 CFR 77.1.(c)
Key factors: study sample, sites, population
19EvidenceFactors
STRONG
THEORY
EVIDENCE OF PROMISE
MODERATE EVIDENCE OF
EFFECTIVENESS
STRONG EVIDENCE OF EFFECTIVENESS
Number of Studies n/a (logic model only) At least one At least one At least one At least one At least two
Study Findings on a Relevant Outcome n/a
Statistically significant, substantively important (0.25 standard deviation or
larger) positive association
Statistically significant positive impact with no unfavorable and overriding impacts
Statistically significant positive
impact with no unfavorable and
overriding impacts
Statistically significant positive
impact with no unfavorable and
overriding impacts
Statistically significant positive
impact with no unfavorable and
overriding impacts
What Works Clearinghouse Standards
n/a n/aMeets without reservations
(RCT)
Meets with reservations(RCT or QED)
Meets without reservations
(RCT)
Meets with reservations(RCT or QED)
Study Sample Size n/a n/a Large sample Large sample Large sample
Number of Study Sites n/a n/a Multi-site sample Multi-site sample Multi-site sample
Similarity of Population n/a n/a
Overlaps with proposed populations
or settings
Overlaps with proposed populations
or settings
Overlaps with proposed
populations and settings
Overlaps with proposed
populations and settings
Official definitions for each term are available in 34 CFR 77.1.(c)
Where to learn more
• WWC Web site (http://ies.ed.gov/ncee/wwc/webinar.aspx), go to “Tools and References“ under the Resources tab to find:
► Webinars about strong studies► Review protocols► Standards Briefs – more on specific WWC standards
• FAQ on designing strong QEDs (http://www.dir-online.com/wp-content/uploads/2015/11/Designing-and-Conducting-Strong-Quasi-Experiments-in-Education-Version-2.pdf
20
THANK YOU!
21