AAI

Boston

May 5, 2012

National Institutes of HealthU.S. Department of Health and Human Services

NIH Peer Review:Where are we and where are we going?

Richard Nakamura, PhDActing Director, Center for Scientific Review, NIH

National Institutes of Health

Much of the biomedical research in the United States is supported by the Federal Government, primarily the National Institutes of Health (NIH)

National Institutes of Health

National Instituteon Alcohol Abuseand Alcoholism

National Instituteon Alcohol Abuseand Alcoholism

National Instituteof Arthritis and

Musculoskeletaland Skin Diseases

National Instituteof Arthritis and

Musculoskeletaland Skin Diseases

National CancerInstitute

National CancerInstitute

National Instituteon Drug Abuse

National Instituteon Drug Abuse

National Instituteof Environmental Health Sciences

National Instituteof Environmental Health Sciences

National Instituteon Aging

National Instituteon Aging

Eunice KennedyShriver National Institute

of Child Health andHuman Development

Eunice KennedyShriver National Institute

of Child Health andHuman Development

National Institute onDeafness and Other

CommunicationDisorders

National Institute onDeafness and Other

CommunicationDisorders

National EyeInstitute

National EyeInstitute

National HumanGenome Research

Institute

National HumanGenome Research

Institute

National Instituteof Mental Health

National Instituteof Mental Health

National Instituteof NeurologicalDisorders and

Stroke

National Instituteof NeurologicalDisorders and

Stroke

National Instituteof General

Medical Sciences

National Instituteof General

Medical Sciences

National Instituteof Nursing Research

National Instituteof Nursing Research

National Libraryof Medicine

National Libraryof Medicine

Center for Scientific Review

Center for Scientific Review

National Centerfor Complementary

and AlternativeMedicine

National Centerfor Complementary

and AlternativeMedicine

National Instituteof Allergy and

Infectious Diseases

National Instituteof Allergy and

Infectious Diseases

John E. FogartyInternational

Center

John E. FogartyInternational

Center

National Centerfor Advancing

Translational Research

National Centerfor Advancing

Translational Research

Clinical Center

Clinical Center

National Institute on Minority Health andHealth Disparities

National Institute on Minority Health andHealth Disparities

National Institute of Biomedical Imagingand Bioengineering

National Institute of Biomedical Imagingand Bioengineering

Office of the DirectorOffice of the Director

Center for InformationTechnology

Center for InformationTechnology

National Heart,Lung, and Blood

Institute

National Heart,Lung, and Blood

Institute

National Instituteof Dental andCraniofacial

Research

National Instituteof Dental andCraniofacial

Research

National Instituteof Diabetes andDigestive and

Kidney Diseases

National Instituteof Diabetes andDigestive and

Kidney Diseases

Spending at NIH

NIH Extramural & Intramural FundingFY 2012 Enacted: $30.9 Billion

83%

17%

Spending Outside NIH$25.7 B

– Supports over 300,000 Scientists & Research Personnel

– Supports over 2,500 Institutions

– $3.4 B Intramural Research – $1.5 B Research Management & Support – $0.3 B Buildings and Facilities, Other$5.2 B

Basic Research 52.0%Applied Research (Clinical)34.6%

Applied Research

(Other) 10.5%

R&D Facilities 0.3% Training & Overhead2.6%

FY 2011 Percent Distribution of Basic and Clinical Research

1978

1979

1980

1981

1982

1983

1984

1985

1986

1987

1988

1989

1990

1991

1992

1993

1994

1995

1996

1997

1998

1999

2000

2001

2002

2003

2004

2005

2006

2007

2008

2009

2010

2011

2012

2013

5%

10%

15%

20%

25%

30%

35%

40%

45%

Fiscal Year

Su

cce

ss R

ate

Grant Success RatesFY 1978-2013

?

PI Applicants: PI Initiative/RFAs

Peer Review

Applications Study Sections Ranking Percentiling

IC

Strategic Goals/Awards/ Funding

Research

• Outcome Progress (Publications/Citations)

• Public Health

The NIH Peer Review Process

• Receives all NIH applications

• Refers them to NIH Institutes/Centers and to scientific review groups

• Reviews majority of grant applications for scientific merit

Your Application Goes to the NIH Center for Scientific Review (CSR)

Focal Point for Initial Review at NIH

CSR Peer Review – Fiscal Year 2011

• 85,000 applications received

• 58,000 applications reviewed at CSR

• 16,000 reviewers

• 230 Scientific Review Officers

• 1,465 review meetings

Enhancing Peer Review

1. Improving Study Section Alignment

• Input from the community

• Internal IRG reviews

• Open Houses

• Advisory Council

Source: Compiled by Science-Metrix from Medline,

NIH data, and Scopus

General Map of Health Research

The Network of CSR Study Sections

0%

1%

2%

3%

4%

5%

6%

7%

PIs in 1980

Per

cen

t o

f P

Is2. Addressing Review and Funding for New Investigators

Projection of Age Distribution of NIH RPG Investigators: 2020

25 30 35 40 45 50 55 60 65 70 75 80 85 90

Age

2. Addressing Review and Funding for New Investigators

NIH decided to control the process at the level of funding rather than during review. The only difference in review is that all the applications from New Investigators are now clustered for discussion

Different paylines for New Investigators and Early Stage Investigators.

2. Funding New Investigators

0.0%

5.0%

10.0%

15.0%

20.0%

25.0%

30.0%

35.0%

40.0%

45.0%

0

1,000

2,000

3,000

4,000

5,000

6,000

1962 1965 1968 1971 1974 1977 1980 1981 1984 1987 1992 1995 1998 2001 2004 2007 2010 est

New

Inve

stiga

tors

as a

Per

cent

age

of A

ll Co

mpe

ting A

war

dees

Num

ber o

f New

and

Expe

rienc

ed R

01 E

quiv

alen

t Prin

cipa

l Inv

estig

ator

s

Fiscal Year

New and Experienced Investigators on R01 Equivalent Grants and New Investigators as a Percentage of All Competing R01 Awardees

(FY 1962 - 2010) preliminary

Established Investigators New Investigators Percent New

3. Advancing Additional Review Platforms

• Additional Review Platforms Help Recruit Reviewers

• Electronic Review Modes Reduce Travel

• Electronic ReviewTelephone Assisted Meetings

Video Assisted Discussions

Internet Assisted Meetings

Telepresence Meetings

3. Telepresence Study Sections

4. Funding the Most Promising Research Earlier

Months: Submission to Posting Critiques

1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 20080%

10%

20%

30%

40%

50%

60%

70%

Per

cen

t o

f T

ota

l A

war

ds

Funding the Most Promising Research Earlier

19982000

20022004

20062008

20100%

10%

20%

30%

40%

50%

60%

70%

Percent of R01 Equivalent Awards by Amendment through 2011

Pe

rce

nt

of

To

tal

Aw

ard

s

A0

A1

A2

4. Funding the Most Promising Research Earlier

5. Recruiting the Best ReviewersSome Successful Strategies

• Move a meeting a year to the West Coast

• Additional review platforms

• Develop a national registry of volunteer reviewers

Searchable database with 5,000 reviewers

• Provide tangible rewards for reviewers

No submission deadlines for chartered members of study sections (effective February 2008)

• Provide flexible time for reviewers

Choice of 3 times/year for 4 years

or 2 times/year for 6 years

6. Focusing More on Impact and Significance and Less on Approach

• Shorten Applications• Scoring Significance• Discussed applications receive additional overall

impact score• Training of Reviewers and Chairs

7. Saving Reviewers Time

• Shorter Applications• Bullet Critiques• Additional Review Platforms

8. Scoring

Impact Score Descriptor

High Impact

1 Exceptional

2 Outstanding

3 Excellent

Moderate Impact

4 Very Good

5 Good

6 Satisfactory

Low Impact

7 Fair

8 Marginal

9 Poor

8. ScoringPriority Scores of R01 and R21 Reviewed by CSR

10 20 30 40 50 60 70 80 900%

10%

20%

30%

40%

50%

60%

Priority Score

Cu

mu

lati

ve

Pe

rce

nt

100 150 200 250 300 350 4000%

10%

20%

30%

40%

50%

60%

Priority Score

Cu

mu

lati

ve

Pe

rce

nt

June 2008

June 2009

9. Continuously Reviewing the Changes

• 12/09 Applicant and Reviewers Survey (64% response)

• 1/10 Advisory Council Survey (291 responses)

• 5/11 Planned Survey on Shorter Applications

New Investigators

• If you submit an R01 grant application

• If you are a New Investigator or Early Stage Investigator

• If NIH has correct info on your career stage

Your Career Stage Is Considered

• New Investigator (NI)PD/PI who has not yet competed successfully for a substantial NIH research grant

• Early Stage Investigator (ESI) PD/PI who qualifies as a New Investigator AND is within 10 years of completing the terminal research degree or is within 10 years of completing medical residency (or equivalent)

http://grants.nih.gov/grants/new_investigators/

New and Early Stage Investigators

Early Career Reviewer Program

Purpose of the Early Career Reviewer (ECR) Program

• To train qualified scientists without significant prior review experience so that they become effective reviewers

• To help emerging researchers advance their careers by exposing them to review experience

• To enrich the existing pool of NIH reviewers by including scientists from less research-intensive institutions as well as those from research-intensive institutions.

Requirements for Being an ECR

• Has not reviewed for NIH beyond one mail review

• Demonstrates scientific qualifications, such as:

A faculty appointment or equivalent

An active independent research program and recent publications in good research journals

• Does not have R01 NIH or equivalent funding

Apply to the ECR Program

• Send a current CV and a list of terms that describe your scientific expertise to:

CSREarlyCareerReviewer@mail.nih.gov

• If eligible, your name will be placed into our ECR database

• You will be invited to serve as an ECR when your expertise is needed for particular applications

• Better distribution of applications across study sections

• Predictive evolution of study sections

• Better applicant tools for requesting study sections

• Better SRO tools for finding reviewers

• Increased diversity and reduced award disparities

• Develop a science of peer review

The Future

This is CSR

September 2009

Questions? Comments?

CSRDirector@csr.nih.gov

What is the Difference Between Significance and Impact?

Significance addresses:

• Does the project address an important problem or a critical barrier to progress in the field?

• If the aims are achieved, how will scientific knowledge, technical capability, and/or clinical practice be improved?

Impact addresses:

• Probability of whether the research will exert a sustained, powerful influence on the research field.

Determination of New Application Status Following Unsuccessful A1

A new application is expected to be substantially different in content and scope

o substantial changes in all sections of the Research Plan, particularly in the Specific Aims and the Research Strategy sections

o fundamental changes in the questions being asked and/or the outcomes examined

o Changes to the Research Plan should produce a significant change in direction and approach for the research project

Determination of New Application Status: No No's….

• Rewording of the Title and Specific Aims does not constitute substantial changes in scope, direction or content.

• Requests for review by a different review committee or funding consideration by a different NIH Institute are not sufficient reasons to consider an application as new.

• Submission to a different FOA is also not sufficient to make an application new.

Range Fold Mean

Average no. Papers

0.8 -25 29 9

Average Citations/

paper

2.5 -55 21 23

SIM= study section impact

metric

2-50 25 24

Statistics for 144 Study Sections

If group bibliometric differences reflect quality differences, can we determine the sources of

the variance?

What is the relative contribution of the quality of the reviewers and the quality of the

applications?

• Announces NIH Scientific Initiatives

• Provides NIH Policy and Administrative Information

• Supplies links to application forms

• Available on the NIH Web Site: http://www.nih.gov

NIH GuideFor Grants and ContractsU.S. Department of Health and Human Services

A Window to Your Application: eRA Commons

eRA Commons is an online interface where a grant applicant can:

• Check submitted grant application for errors and warnings and view final image

• Track review assignment, view review outcomes (score, summary statements), find contact info

• Update Personal Profile to ensure Early Stage Investigator eligibility is in place

• Submit pre-award information (just in time)• View Notice of Award and other key documents

And much more! https://commons.era.nih.gov/commons/

View the Videos

• NIH Peer Review Revealed

• NIH Tips for Applicants

• What Happens to Your NIH Grant Application

http://www.csr.nih.gov/video/video.asp

Recruiting the Best Reviewers Academic Rank of ALL CSR Reviewers

2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 20110%

10%

20%

30%

40%

50%

60%

70%

80%

PROFESSOR ASSOCIATE PROFESSOR ASSISTANT PROFESSOR

Recruiting the Best Reviewers Academic Rank of CSR Standing Reviewers

2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 20110%

10%

20%

30%

40%

50%

60%

70%

80%

PROFESSOR ASSOCIATE PROFESSOR ASSISTANT PROFESSOR

CSR Mission Statement

To see that NIH grant applications receive fair, independent, expert, and timely reviews – free from inappropriate influences – so NIH can fund the most promising research.

Longer Term Optimization:

Structure of CSR

• Current Map of Science- Pubmed & Scopus

• Analyze the network of study sections

• Benefits

Inform redesign of study section network

Optimize referral of applications to study sections

Improve selection of reviewers

Improve staff selection and interactions

Evaluating the Anatomy of CSR