Evidence Reviews - Urban Institute · Enter “Evidence Reviews” Evidence reviews, and in...

34
Evidence Reviews Jacob Alex Klerman Senior Fellow, Abt Associates Editor, Evaluation Review Federal Evaluation and Evidence Workshop #2 Urban Institute, September 13, 2017

Transcript of Evidence Reviews - Urban Institute · Enter “Evidence Reviews” Evidence reviews, and in...

Page 1: Evidence Reviews - Urban Institute · Enter “Evidence Reviews” Evidence reviews, and in particular federally funded evidence reviews, are perhaps the most exciting and most important

Evidence Reviews

Jacob Alex Klerman

Senior Fellow, Abt Associates

Editor, Evaluation Review

Federal Evaluation and Evidence Workshop #2

Urban Institute, September 13, 2017

Page 2: Evidence Reviews - Urban Institute · Enter “Evidence Reviews” Evidence reviews, and in particular federally funded evidence reviews, are perhaps the most exciting and most important

Abt Associates | pg 2

Outline

▪ Overview of Evidence Reviews

▪ Four Steps in Evidence Reviews

▪ Effect of Evidence Reviews

▪ Three Open Issues in Evidence Reviews

Page 3: Evidence Reviews - Urban Institute · Enter “Evidence Reviews” Evidence reviews, and in particular federally funded evidence reviews, are perhaps the most exciting and most important

Abt Associates | pg 3

Outline

▪ Overview of Evidence Reviews

▪ Four Steps in Evidence Reviews

▪ Effect of Evidence Reviews

▪ Three Open Issues in Evidence Reviews

Page 4: Evidence Reviews - Urban Institute · Enter “Evidence Reviews” Evidence reviews, and in particular federally funded evidence reviews, are perhaps the most exciting and most important

Abt Associates | pg 4

Evidence Based Policy Paradigm

Research Evidence

Better Policies and Programs

Better Outcomes for Clients

Page 5: Evidence Reviews - Urban Institute · Enter “Evidence Reviews” Evidence reviews, and in particular federally funded evidence reviews, are perhaps the most exciting and most important

Abt Associates | pg 5

How Does Research Evidence Affect

Policies and Programs?

Research Evidence

Missing Step!!!

Better Policies and Programs

Better Outcomes for Clients

▪ Translating research evidence into

correct and accessible guidance for

decision makers requires …

– People with a range of skills

– Considerable resources

Page 6: Evidence Reviews - Urban Institute · Enter “Evidence Reviews” Evidence reviews, and in particular federally funded evidence reviews, are perhaps the most exciting and most important

Abt Associates | pg 6

Enter “Evidence Reviews”

Research Evidence

EvidenceReview

Better Policies and Programs

Better Outcomes for Clients

▪ Translating research evidence into

correct and accessible guidance for

decision makers requires …

– People with a range of skills

– Considerable resources

Page 7: Evidence Reviews - Urban Institute · Enter “Evidence Reviews” Evidence reviews, and in particular federally funded evidence reviews, are perhaps the most exciting and most important

Abt Associates | pg 7

Outline

▪ Overview of Evidence Reviews

▪ Four Steps in Evidence Reviews

▪ Effect of Evidence Reviews

▪ Three Open Issues in Evidence Reviews

Page 8: Evidence Reviews - Urban Institute · Enter “Evidence Reviews” Evidence reviews, and in particular federally funded evidence reviews, are perhaps the most exciting and most important

Abt Associates | pg 8

Step 1: Identify All Studies

Accessible

Product

Live

Guidance

Synthesis

Study A

Study B

Study C

Study D

Identify

All

Studies

Assess

Quality of

Individual

Studies

Combine

Evidence from

Multiple Studies

Disseminate Results

to Decision Makers

Page 9: Evidence Reviews - Urban Institute · Enter “Evidence Reviews” Evidence reviews, and in particular federally funded evidence reviews, are perhaps the most exciting and most important

Abt Associates | pg 9

Step 2: Assess Quality of Individual Studies

Accessible

Product

Live

Guidance

Synthesis

Study A

Study B

Study C

Study D

Identify

All

Studies

Assess

Quality of

Individual

Studies

Combine

Evidence from

Multiple Studies

Disseminate Results

to Decision Makers

Page 10: Evidence Reviews - Urban Institute · Enter “Evidence Reviews” Evidence reviews, and in particular federally funded evidence reviews, are perhaps the most exciting and most important

Abt Associates | pg 10

Step 3: Combine Evidence

from Multiple Studies

Accessible

Product

Live

Guidance

Synthesis

Study A

Study B

Study C

Study D

Identify

All

Studies

Assess

Quality of

Individual

Studies

Combine

Evidence from

Multiple Studies

Disseminate Results

to Decision Makers

Page 11: Evidence Reviews - Urban Institute · Enter “Evidence Reviews” Evidence reviews, and in particular federally funded evidence reviews, are perhaps the most exciting and most important

Abt Associates | pg 11

Step 4: Disseminate Results

to Decision Makers

Accessible

Product

Live

Guidance

Synthesis

Study A

Study B

Study C

Study D

Identify

All

Studies

Assess

Quality of

Individual

Studies

Combine

Evidence from

Multiple Studies

Disseminate Results

to Decision Makers

Page 12: Evidence Reviews - Urban Institute · Enter “Evidence Reviews” Evidence reviews, and in particular federally funded evidence reviews, are perhaps the most exciting and most important

Abt Associates | pg 12

Outline

▪ Overview of Evidence Reviews

▪ Four Steps in Evidence Reviews

▪ Effect of Evidence Reviews

▪ Three Open Issues in Evidence Reviews

Page 13: Evidence Reviews - Urban Institute · Enter “Evidence Reviews” Evidence reviews, and in particular federally funded evidence reviews, are perhaps the most exciting and most important

Abt Associates | pg 13

Direct Effect

Much more likely if an Evidence Review

identifies program models and disseminates that information

Research Evidence

EvidenceReview

Better Policies and

Programs

Better Outcomes for Clients

Page 14: Evidence Reviews - Urban Institute · Enter “Evidence Reviews” Evidence reviews, and in particular federally funded evidence reviews, are perhaps the most exciting and most important

Abt Associates | pg 14

Indirect Effect

Positive Feedback Loop:

Evidence Review’s induce better evaluations:

disseminating best practices, inducing Evaluation TA

Research Evidence

EvidenceReview

Better Policies and Programs

Better Outcomes for Clients

Page 15: Evidence Reviews - Urban Institute · Enter “Evidence Reviews” Evidence reviews, and in particular federally funded evidence reviews, are perhaps the most exciting and most important

Abt Associates | pg 15

Outline

▪ Overview of Evidence Reviews

▪ Four Steps in Evidence Reviews

▪ Effect of Evidence Reviews

▪ Three Open Issues in Evidence Reviews

Page 16: Evidence Reviews - Urban Institute · Enter “Evidence Reviews” Evidence reviews, and in particular federally funded evidence reviews, are perhaps the most exciting and most important

Abt Associates | pg 16

Issue 1: The Evidence Base

▪ Foundation of Evidence Reviews: The individual studies

▪ But, there simply aren’t enough studies

▪ One single positive result is only weak evidence

– Efficacy vs. effectiveness

– Replicability crisis

– Generalization to other environments

▪ (potential for) Negative Feedback Loop: Once

“demonstrated effective,” another/more evaluation(s) can

only hurt the program

There is no substitute for replication; even though it’s

expensive, long time-line, and unglamorous

Page 17: Evidence Reviews - Urban Institute · Enter “Evidence Reviews” Evidence reviews, and in particular federally funded evidence reviews, are perhaps the most exciting and most important

Abt Associates | pg 17

Issue 2: Synthesizing the Evidence

▪ Some Evidence Reviews are using simple heuristics to

synthesize the evidence; e.g.,

– One significant positive impact (in any domain)

– Two significant positive impacts and no significant negative impacts

– Simple mean of impacts

▪ Need to move towards formal “meta-analysis”;

a formal statistical approach for

– Combining estimates across studies, and

– Exploring when and how impact varies

But meta-analysis requires 3+ studies …

Page 18: Evidence Reviews - Urban Institute · Enter “Evidence Reviews” Evidence reviews, and in particular federally funded evidence reviews, are perhaps the most exciting and most important

Abt Associates | pg 18

Issue 3: External Validity

▪ Suppose a site has a choice between two program models

▪ Should it choose the program model with

1. Strongest (most positive) overall evidence? or

2. Evidence of effectiveness on a population “most similar” to ours?

▪ Recent academic work and efforts of evidence reviews

leans strongly towards #2/“most similar”

▪ My (personal) position: That’s a serious mistake

Page 19: Evidence Reviews - Urban Institute · Enter “Evidence Reviews” Evidence reviews, and in particular federally funded evidence reviews, are perhaps the most exciting and most important

Abt Associates | pg 19

My (personal) Position

▪ Science/policy analysis/evidence-based policy making is

impossible if we don’t think results generalize; so …

▪ Assume results generalize and choose the program with

the strongest overall evidence, unless

– Obviously implausible, or

– Clear evidence that impacts are different for this population

▪ For demonstrated effective programs, move aggressively

to explore external validity

– Meta-analysis of existing studies

– New studies focused on external validity; i.e., how results vary

– Caveat: Requires huge samples;

perhaps 5x cases required to demonstrate effectiveness

Page 20: Evidence Reviews - Urban Institute · Enter “Evidence Reviews” Evidence reviews, and in particular federally funded evidence reviews, are perhaps the most exciting and most important

Abt Associates | pg 20

Outline

▪ Overview of Evidence Reviews

▪ Four Steps in Evidence Reviews

▪ Effect of Evidence Reviews

▪ Three Open Issues in Evidence Reviews

Page 21: Evidence Reviews - Urban Institute · Enter “Evidence Reviews” Evidence reviews, and in particular federally funded evidence reviews, are perhaps the most exciting and most important

Abt Associates | pg 21

Research Evidence

“Evidence Review”

Better Policies and Programs

Better Outcomes for Clients

Page 22: Evidence Reviews - Urban Institute · Enter “Evidence Reviews” Evidence reviews, and in particular federally funded evidence reviews, are perhaps the most exciting and most important

Evidence Reviews

Jacob Alex Klerman, Abt Associates

[email protected]

Feel free to contact me if you want copies of the papers

(or just to talk more about these issues)

Page 23: Evidence Reviews - Urban Institute · Enter “Evidence Reviews” Evidence reviews, and in particular federally funded evidence reviews, are perhaps the most exciting and most important

Abt Associates | pg 23

For More on these Issues …

▪ Three 2017 issues of Evaluation Review on Evidence

Reviews– For an overview, my “Special Issue Editor’s Overview Essay”

▪ Also two 2017 issues of Evaluation Review on “External

Validity in Evidence Reviews”– On external validity: my “Editors Comment

Page 24: Evidence Reviews - Urban Institute · Enter “Evidence Reviews” Evidence reviews, and in particular federally funded evidence reviews, are perhaps the most exciting and most important

Abt Associates | pg 24

For More on these Issues …

▪ On the Evidence Review movement: TR Westbrook, SA

Avellar, and Neil Seftor. 2017. “Reviewing the Reviews:

Examining Similarities and Differences Between Federally

Funded Evidence Reviews.” Evaluation Review.

▪ On meta-analysis for Evidence Reviews: JC Valentine, et

al. 2017. “Synthesizing Evidence in Public Policy Contexts:

The Challenge of Synthesis when there are Only a Few

Studies”.” Evaluation Review.

▪ On positive feedback loops: N. Seftor. 2017. “Raising The

Bar”. Evaluation Review.

Page 25: Evidence Reviews - Urban Institute · Enter “Evidence Reviews” Evidence reviews, and in particular federally funded evidence reviews, are perhaps the most exciting and most important

Abt Associates | pg 25

Parting Thoughts

▪ Open Call for Papers for Special Issue of Evaluation

Review on Evaluation Technical Assistance– Looking for federal perspectives

– http://journals.sagepub.com/pb-

assets/cmscontent/ERX/ERX_SI_EvalTechAssist2017.pdf

▪ For copies of any of these papers, shoot me an email:

[email protected]

Page 26: Evidence Reviews - Urban Institute · Enter “Evidence Reviews” Evidence reviews, and in particular federally funded evidence reviews, are perhaps the most exciting and most important

Back Up Slides

Page 27: Evidence Reviews - Urban Institute · Enter “Evidence Reviews” Evidence reviews, and in particular federally funded evidence reviews, are perhaps the most exciting and most important

Abt Associates | pg 27

Outline

▪ Overview of Evidence Reviews

▪ Four Steps in Evidence Reviews

▪ Effect of Evidence Reviews

▪ Three Open Issues in Evidence Reviews

Page 28: Evidence Reviews - Urban Institute · Enter “Evidence Reviews” Evidence reviews, and in particular federally funded evidence reviews, are perhaps the most exciting and most important

Abt Associates | pg 28

Challenge: Combining Across Studies

With flimsy justification, and in small type buried in routine documents,

the Trump administration has informed 81 local governments and health

groups that it will end grants they have received to run teen pregnancy

prevention programs, two years before the grants are scheduled to end.

The decision is unsettling even by the disquieting standards of this anti-

science administration.

Page 29: Evidence Reviews - Urban Institute · Enter “Evidence Reviews” Evidence reviews, and in particular federally funded evidence reviews, are perhaps the most exciting and most important

Abt Associates | pg 29

(Federally Funded) Evidence Reviews

Evidence reviews are designed to identify

evaluations, assess their quality, and summarize

evidence into user-friendly materials free or

research jargon.

”TR Westbrook, SA Avellar, and Neil Seftor. 2017.

“Reviewing the Reviews: Examining Similarities and Differences Between

Federally Funded Evidence Reviews.” Evaluation Review.

Page 30: Evidence Reviews - Urban Institute · Enter “Evidence Reviews” Evidence reviews, and in particular federally funded evidence reviews, are perhaps the most exciting and most important

Abt Associates | pg 30

Some Federally Funded Evidence Reviews

Funder Name

DHHS/ACF Employment Strategies for Low-Income

Adults Evidence Review

DHHS/ACF Home Visiting Evidence of Effectiveness

DHHS/AHRQ Evidence Based Practice Center

(health care)

DHHS/ASPE Teen Pregnancy Prevention Review

DOJ Crime Solutions

ED/IES What Works Clearinghouse

DHHS/SAMHSA National Registry of Evidence-Based

Programs and Practices

(mental health and subst. abuse disorders)

Page 31: Evidence Reviews - Urban Institute · Enter “Evidence Reviews” Evidence reviews, and in particular federally funded evidence reviews, are perhaps the most exciting and most important

Abt Associates | pg 31

Enter “Evidence Reviews”

Evidence reviews, and in particular federally

funded evidence reviews, are perhaps the most

exciting and most important developments in

“evidence-based” policy-making —that is, the use

of research evidence in deciding what programs

and policies to implement and the details of those

programs and policies.

” JA Klerman. 2017.

“Special Issue Editor’s Overview Essay.” Evaluation Review.

Page 32: Evidence Reviews - Urban Institute · Enter “Evidence Reviews” Evidence reviews, and in particular federally funded evidence reviews, are perhaps the most exciting and most important

Abt Associates | pg 32

Often a Major Effort

▪ Person months

▪ By skilled scholars

– In judging individual studies

– In combining evidence from multiple studies

– In disseminating findings to non-technical audiences

▪ Applying common and vetted standards

Certainly more than we can expect of

practitioners, decision makers, and most scholars

Page 33: Evidence Reviews - Urban Institute · Enter “Evidence Reviews” Evidence reviews, and in particular federally funded evidence reviews, are perhaps the most exciting and most important

Abt Associates | pg 33

Direct Effect

▪ National programs adopt “higher impact program models”

▪ National funding streams require or encourage (e.g., give

additional points for) grantees to adopt “demonstrated

effective program models”

▪ Local programs adopt “higher impact program models”

Much more likely if an Evidence Review

identifies program models and disseminates that information

Page 34: Evidence Reviews - Urban Institute · Enter “Evidence Reviews” Evidence reviews, and in particular federally funded evidence reviews, are perhaps the most exciting and most important

Abt Associates | pg 34

Issue 3: External Validity

▪ Formally, this in invalid!

▪ Worrying some scholars

a lot

– See two volume Special

Issue of Evaluation

Review

▪ Pushing some

Evidence Reviews to

emphasize evidence for

“similar populations”

– Rather than overall best

program

Evaluate in Denver

Implement in Atlanta