ENHANCING PEER REVIEWENHANCING PEER REVIEW What Reviewers Need to Know NowWhat Reviewers Need to Know Now

Slides Accompanying Video of Dr. Alan Willard, March Slides Accompanying Video of Dr. Alan Willard, March 2009 2009

1

Prepare for the May/June Review Meetings

Supplement resources on the Enhancing Peer Review Web site:

http://enhancing-peer-review.nih.gov

Video – Overview of EnhancementsVideo – Overview of Enhancements Frequently Asked Questions Frequently Asked Questions E-mailbox for QuestionsE-mailbox for Questions

Goals of this VideoGoals of this Video

2

Changes to ReviewChanges to ReviewBeginning at May/June 2009 MeetingsBeginning at May/June 2009 Meetings Enhanced Review Criteria

– Significance, Investigator(s), Innovation, Approach, Environment Templates for Structured Critiques Scoring of Individual Review Criteria New 1-9 Scoring Scale

3

Goals of the ChangesGoals of the Changes

Clearer understanding of the basis of application ratings

More emphasis on impact and less emphasis on technical details

Succinct, well-focused critiques that evaluate, rather than describe, applications

Routine use of the entire rating scale

4

Before the Review MeetingBefore the Review Meeting

When reading applications the assigned reviewers should: Address all applicable criteria and other

review considerations Identify major strengths and weaknesses Assign scores to each of the 5 “core”

criteria Assign an overall impact/priority score

5

Preparation of CritiquesPreparation of Critiques

When writing your critiques: Use bulleted points to make succinct,

focused comments Short narratives may occasionally be

appropriate, but should be rare Focus on major strengths and

weaknesses (ones that impacted your overall rating of the application)

6

Features of Critique TemplatesFeatures of Critique Templates

Boxes for evaluating:– Each core review criterion– Other applicable review criteria and

considerations– Overall impact of the application

A box for “advice to applicants” Hyperlinks to web pages providing

descriptions of review criteria and additional review considerations

7

Excerpt from a Critique TemplateExcerpt from a Critique Template

List major strengths and weaknesses List major strengths and weaknesses that influenced the overall that influenced the overall impact/priority score impact/priority score

Limit text to ¼ page per criterion, Limit text to ¼ page per criterion, although more text may occasionally be although more text may occasionally be neededneeded 8

Scoring of Individual Review CriteriaScoring of Individual Review Criteria

There are 5 “core” criteria for most types of grant applications

For example, the core criteria for R01s are:– Significance– Investigator(s) – Innovation– Approach– Environment

Use the 9-point scale (1 = exceptional, 9 = poor) for the five “core” review criteria.

9

Overall Impact/Priority ScoresOverall Impact/Priority Scores

Consider criterion strengths and weaknesses of each application in determining an overall impact/priority score

Recognize this is a NEW scoring system and focus on the guidelines for its use

This new scoring system is intended to reflect the “real-world” range of the quality of applications typically seen in actual study sections

It is ESSENTIAL that reviewers take advantage of this unique opportunity to use the entire 1-9 range

10

Scoring DescriptionsScoring DescriptionsImpact Score Descriptor Additional Guidance on

Strengths/Weaknesses

High

1 Exceptional Exceptionally strong with essentially no weaknesses

2 Outstanding Extremely strong with negligible weaknesses

3 Excellent Very strong with only some minor weaknesses

Medium

4 Very Good Strong but with numerous minor weaknesses

5 Good Strong but with at least one moderate weakness

6 Satisfactory Some strengths but also some moderate weaknesses

Low

7 Fair Some strengths but with at least one major weakness

8 Marginal A few strengths and a few major weaknesses

9 Poor Very few strengths and numerous major weaknesses

Non-numeric score options: NR = Not Recommended for Further Consideration, DF = Deferred, AB = Abstention, CF = Conflict, NP = Not Present, ND = Not DiscussedMinor Weakness: An easily addressable weakness that does not substantially lessen impactModerate Weakness: A weakness that lessens impactMajor Weakness: A weakness that severely limits impact 11

Before Attending the Review MeetingBefore Attending the Review Meeting

Post critiques to the Internet-Assisted Review (IAR) Web site

Enter criterion scores and overall/priority score in IAR

IAR tutorials and documentation are available on the Enhancing Peer Review Web site

12

Procedure for Discussed ApplicationsProcedure for Discussed Applications

Assigned reviewers will discuss strengths and weaknesses of each application– Recommend overall impact/priority score– Criterion scores will not be discussed by the

committee All eligible members will record an

overall impact/priority score (as is presently true)

13

After the Review MeetingAfter the Review Meeting

To justify the overall impact/priority score:

Assigned reviewers whose opinions changed as a result of discussion at the meeting should use IAR: – To modify their criterion scores– To post revised critiques

14

Summary StatementsSummary Statements

Overall impact/priority scores of discussed applications will be the mean of scores voted by all eligible reviewers, multiplied by 10

Final scores will range from 10-90, in whole numbers

Summary statements for ALL applications will include the criterion scores and critiques posted by assigned reviewers

15

For additional information:For additional information:

Enhancing Peer Review at NIH Web SiteEnhancing Peer Review at NIH Web Site

http://enhancing-peer-review.nih.gov

Thank you for your review service Thank you for your review service

16