NIH Peer Review of Small Business Applications

11th NIH SBIR/STTR Conference

July 2009

National Institutes of HealthU.S. Department of Health and Human Services

Dana Jeffrey Plude, Ph.D.Acting DirectorDivision of AIDS, Behavioral and Population Sciences

NIH DUAL PEER REVIEW• Scientific Review Group

Evaluate scientific and technical meritRecommend level of support, durationDoes not make funding decisions

• Advisory CouncilReviews the reviewFunding recommendations to Institute/CenterEvaluate priorities/relevanceAdvise on Policy

• Funding decision made by Institute/Center

2

Translational and Translational and

Clinical SciencesClinical Sciences

Cardiovascular and Cardiovascular and

Respiratory Sciences Respiratory Sciences

Surgical Sciences, Surgical Sciences,

Biomedical Biomedical

Imaging and Imaging and

Bioengineering Bioengineering

Musculoskeletal, Oral Musculoskeletal, Oral

And Skin Sciences And Skin Sciences

Oncology 2 – Oncology 2 –

Translational Clinical Translational Clinical

Vascular and Vascular and

HematologyHematology

Physiological and Physiological and

Pathological Sciences Pathological Sciences

Endocrinology, Endocrinology,

Metabolism, Metabolism,

Nutrition &Nutrition &

Reproductive SciencesReproductive Sciences

ImmunologyImmunology

Infectious DiseasesInfectious Diseases

& Microbiology& Microbiology

Digestive, Kidney &Digestive, Kidney &

Urological Systems Urological Systems

Neuroscience, Development Neuroscience, Development

and Agingand Aging

Brain Disorders &Brain Disorders &

Clinical NeuroscienceClinical Neuroscience

Molecular, Cellular &Molecular, Cellular &Developmental NeurosciencDevelopmental Neuroscience

Integrative, Functional & Integrative, Functional &

Cognitive NeuroscienceCognitive Neuroscience

Emerging Technologies &Emerging Technologies &

Training in NeuroscienceTraining in Neuroscience

Biology of Development Biology of Development

& Aging& Aging

Biobehavioral &Biobehavioral &

Behavioral ProcessesBehavioral Processes

Risk, Prevention& Risk, Prevention&

Health Behaviors Health Behaviors

Population Sciences & Population Sciences &

EpidemiologyEpidemiology

Healthcare Delivery Healthcare Delivery

& Methodologies& Methodologies

AIDS &AIDS &

Related ResearchRelated Research

AIDS, Behavioral AIDS, Behavioral

and Population Sciencesand Population SciencesBasic and Integrative Basic and Integrative

Biological SciencesBiological Sciences

Biological Chemistry & Biological Chemistry &

Macromolecular Macromolecular

Biophysics Biophysics

Bioengineering SciencesBioengineering Sciences

& Technologies& Technologies

Genes, Genomes Genes, Genomes

& Genetics & Genetics

Oncology 1 – Basic Oncology 1 – Basic

TranslationalTranslational

Cell BiologyCell Biology

Interdisciplinary Interdisciplinary

Molecular Molecular

& Training& Training

CSR Review DivisionsCSR Review Divisions

PEER REVIEW IN CSR• Scientific Review Groups/Study Sections/Review

Committees/Special Emphasis Panels - Small Business applications are reviewed in Special Emphasis Panels.

• Scientific Review Officer is responsible for the management of the meeting.

• Chair and 10 - 24 members who are from academia and small businesses

• 40 - 100 applications are reviewed at each study section meeting (face to face meeting).

• CSR is experimenting with review platforms – video conferences, asynchronous extended discussion, and others.

4

What It Looks Like: Video Enhanced Discussions

What It Looks Like: Asynchronous Electronic

Discussions

SCIENTIFIC REVIEW OFFICER

• Performs administrative and technical review of applications to ensure completeness

• Selects reviewers based on broad input• Assigns reviewers (generally at least two

reviewers and one reader per application)• Manages review meeting/Designated Federal

Official• Prepares Summary Statement• Provides requested information about review

recommendations to Institutes/Centers and advisory councils

7

CRITERIA FOR SELECTION OF PEER REVIEWERS

• Demonstrated scientific expertise• Doctoral degree or equivalent• Mature judgment • Work effectively in a groups• Breadth of perspective• Impartiality• Interest in serving• Adequate representation of women and minority

Scientists• Geographic distribution• Small Business representation

8

REVIEW MEETINGS• Closed to the public• Introductions• SRO instructions: Confidentiality, conflict

of interest, new NIH policies, procedures• Approximately 60% of the applications are

Discussed, the ‘less competitive’ 40% of the applications are Not Discussed

• Applications are discussed in ‘score order’ based on Preliminary “Overall Impact Score” given by the three assigned reviewers

Note - any member can nominate an application for discussion

9

REVIEW MEETINGS (cont).Individual review of top 60% applications

Conflicts excusedAssigned reviewers/readers: preliminary scores, evaluationsGeneral discussion, including human subjects, vertebrate animals, biohazard issuesPrivate assignment of priority scoreDiscussion of budget, data sharing, resource sharing issues

“Mock” Study Section video http://cms.csr.nih.gov/ResourcesforApplicants/InsidetheNIHGrantReviewProcessVideo.htm

10

REVIEW LOGISTICS

• Reviewers receive applications and assignments one to two months in advance of meeting.

• Reviewers post preliminary scores and critiques on a secure web site in advance.

• Reviewers may see other critiques only after their own are posted.

• Are not aiming for consensus but outliers must explain their views

• Critiques are modified in light of discussion.• SRO prepares Resume and Summary of

Discussion.

11

Enhancing Peer ReviewEnhancing Peer Review

““Fund the best science, by the best Fund the best science, by the best scientists, with the least administrative scientists, with the least administrative

burden…” burden…”

Elias Zerhouni, MD, Former Director, Elias Zerhouni, MD, Former Director, NIHNIH

2008: The Year of Peer Review2008: The Year of Peer Review

RecommendationsRecommendations

Amended Applications:Amended Applications:

To speed the funding of meritorious science and To speed the funding of meritorious science and minimize reviewer burden:minimize reviewer burden:

•As of January 25, 2009, all original new applications (i.e., never submitted) and competing renewal applications will be permitted only a single amendment (A1)single amendment (A1).

Balanced and Fair Reviews Across Career Balanced and Fair Reviews Across Career Stages and Scientific FieldsStages and Scientific Fields

• New Investigator: New Investigator: Not previously competed successfully as PD/PI for a significant NIH independent research award. 

• Early Stage Investigator: Early Stage Investigator: Within 10 years of completing terminal research degree or is within 10 years of completing medical residency (or the equivalent). 

• These Designations are IRRELEVANT for Small Business applications – only R01 research grant applications are coded for New PI/ESI

ScoringScoringTo improve the transparency of the scoring process:To improve the transparency of the scoring process:• Score applications on five review criteria using a scale of 1-9five review criteria using a scale of 1-9.

• Preliminary overall impact score using 1-9 scaleoverall impact score using 1-9 scale.

• Discussed applications will receive an overall score from each eligible (i.e., without conflicts of interest) panel member and these scores will be averaged to one decimal place, and multiplied by 10. The 81 possible priority scores will thus range from 10-90.

• Percentiles will be reported in whole numbers.

• All applications will receive scoresAll applications will receive scores::

Not DiscussedNot Discussed applications will receive initial criterion scores from the three assigned reviewers

• Goal: Improve the transparency of the scoring process:

Score applications on five review criteria using a scale of 1-9.

• Preliminary overall impact score using 1-9 scale.

o Should not be the average of the criterion scores.

o Not Discussed applications will receive initial criterion scores from the three assigned reviewers.

Scoring on a 9-Point Scale

Scoring Descriptions

Impact Score Descriptor Strengths/Weaknesses

High Impact

1 Exceptional

 Weaknesses

2 Outstanding

3 Excellent

Moderate Impact

4 Very Good

5 Good

6 Satisfactory

Low Impact

7 Fair

8 Marginal

9 Poor

Scoring

Expectations:

• Reviewers participate in entire meeting .

• To rank the applications in order of review based on preliminary score, all applications must be scored; applications with initial scores of 7-9 are those unlikely to be discussed.

• Once at the meeting, the full range of scores is available to assign an overall impact score for all discussed applications.

Scoring (cont.)

• Criterion scores are considered part of the critique and will not be discussed at the review meeting.

They may be changed in the EDIT phase in IAR.

Clustering

• Phase I, Phase II and Fast Track Applications will be clustered together (provided sufficient numbers).

Phase I: FEASIBILITY

Phase II: EFFICACY

Fast Track: Both Components

Clustering of Clinical Applications

• The goal of clustering clinical applications in review is to increase fairness in the process of reviewing clinical applications.

A clinical application will be defined as human subjects research minus Exemption 4.

Order of Review

• Goal: To discuss applications in order of

average preliminary score.

• Why?

Concern: variation of scores during different times of the meeting.

One recommendation was to recalibrate scores at the end of the meeting.

• The Solution

Recalibrate “dynamically” throughout meeting.

Order of Review

• For calibration purposes…

Begin meeting by discussing the best scored application (any activity code).

o All Activity codes are clustered if f feasible (if at least 10 discussed, may include R41/43, R42/44 and Fast Track as groups that can be clustered).

Order of Review

Summary

• Discussion order is based on the average of the impact scores from assigned reviewers.

• Final scores of discussed applications may differ from preliminary scores as re-calibration happens dynamically.

• Discuss ~ 40% of small business applications.

• SRO will then ask if there are any other applications that panel wishes to discuss.

• The remaining applications will not be discussed (applications receive criterion scores only).

Not Discussed

Final Scores

• Discussed applications will receive an overall score from each eligible (i.e., without conflicts of interest) panel member and these scores will be averaged to one decimal place, and multiplied by 10. The 81 possible priority scores will thus range from 10-90.

• Percentiles will be reported in whole numbers.

•Summary statement will be shorter and more focused.

•Discussed applications will also have a summary of the panel’s discussion at the meeting.

•ALL applications will be scored.

Not discussed applications will receive criterion scores only.

Summary Statements

CritiquesCritiques

To improve the quality of the critiques and to To improve the quality of the critiques and to focus reviewer attention on the review criteria:focus reviewer attention on the review criteria:

•Electronic template for critiques Electronic template for critiques that will prompt for strengths and weaknesses for each criterion.

•Summary statement that will be shortershorter and more focused.

•Discussed applications only also will have a summary of summary of the panel’s discussiondiscussion at the meeting.

•In 2009, ALL applications will be scoredALL applications will be scored.•Not discussed applications will receive criterion scorescriterion scores only in addition to the reviewers’ critiques

Template-Based Critiques• Critique template contains a total of 18 boxes.

Reviewers should provide text for only those criteria that are applicable.

1. Overall Impact 7. Resubmission 13. Biohazards

2. Significance 8. Renewal 14. Budget and Period of Support

3. Investigator(s) 9. Revision 15. Select Agents

4. Innovation 10. Protection of Human Subjects

16. Applications from Foreign Organization

5. Approach 11. Inclusion of Women, Minorities, and Children

17. Resource Sharing Plan

6. Environment 12. Vertebrate Animals 18. Additional Comments to Applicant

Template-Based Critiques

• Goal: Write evaluative statements and to discourage summarizing the application.

Comments should be in the form of bullet points or, if necessary, short narratives.

Do not record scores on the critique template.

The entire template is uploaded to IAR to become part of the summary statement.

1. Significance  Please limit text to ¼ page

Strengths Weaknesses

REVIEW CRITERIASpecifically tailored for small business

applications:

Significance

Investigator(s)

Innovation

Approach

Environment Overall Impact – weigh as appropriate,

consider impact on scientific field

32

SIGNIFICANCE• Does the project have commercial potential to lead to

a marketable product, process or service? • If the aims of the project are achieved, how will

scientific knowledge or clinical practice be advanced? • Does this study address an important problem and

what commercial and societal benefits might be derived from the proposed research?

• What will be the effect of these studies on the concepts, methods, technologies, treatments, services, or preventative interventions that drive this field?

• Does the project lead to enabling technologies (e.g., instrumentation, software) for further discoveries?

• Will the technology have a competitive advantage over existing/alternate technologies that can meet market needs?

33

INVESTIGATORS• Is the PD/PI appropriately trained and

capable of coordinating and managing the proposed project?

• Are the investigators well suited to carry out this work and does the investigative team bring complementary and integrated expertise to the project (if applicable)?

• Is the work proposed appropriate to the experience level of the PD/PI and other researchers, including consultants and subcontractors (if any)?

• Are the relationships of the key personnel to the small business and to other institutions appropriate for the work proposed?

34

INNOVATION• Are the aims original and innovative? • Does the project challenge existing

paradigms or clinical practice; address an innovative hypothesis or critical barrier to progress in the field?

• Does the project develop or employ novel concepts, approaches, methodologies, tools, or technologies for this area?

35

APPROACH• Are the conceptual or clinical framework, design,

methods, and analyses adequately developed, well-integrated, and appropriate to the aims of the project?

• Is the proposed plan a sound approach for establishing technical and commercial feasibility?

• Does the applicant acknowledge potential problem areas and consider alternative strategies?

• Are the milestones and evaluation procedures clearly described and appropriate?

• For applications designating multiple PD/PIs, is the leadership approach, including the designated roles and responsibilities, governance and organizational structure consistent with and justifies by the aims of the project and the expertise of each of the PD/PIs?

36

ENVIRONMENT• Is there sufficient access to resources (e.g.,

equipment, facilities)? • Does the scientific and technological

environment in which the work will be done contribute to the probability of success?

• Do the proposed studies benefit from unique features of the scientific environment, or subject populations, or employ useful collaborative arrangements?

• Is there evidence of institutional support?

37

SPECIAL REVIEW CRITERIA• Resubmissions

Responses to comments from the previous reviewImprovements in application

• Phase II ApplicationsProgress in Phase I – objectives met, feasibility demonstratedCommercialization PlanCommercial potential

• Fast Track ApplicationsMeasurable goals/milestonesCommercialization PlanCommercial potentialFunding commitments/resources

38

ADDITIONAL REVIEW CRITERIA

• Protection of human subjects from research risk

• Inclusion of women, minorities and children

• Care and use of vertebrate animals• Biohazards

Problems in these areas will have a deleterious impact on the review outcome/score

39

OTHER REVIEW CONSIDERATIONS

• Budget – amount, duration• Plans for Data Sharing if required• Plans for Model Organism sharing • Genome Wide Association Studies –

plans for sharing data generated

These do not contribute to the score

40

REVIEW OUTCOME

• Not recommended for further consideration – rare, usually due to serious ethical or safety concerns

• Deferral – also rare, study section does not have sufficient information to make a determination

• Not Discussed 40% (approximately)• Scored 60% (approximately)

41

SUMMARY STATEMENT

Not Discussed/Priority Score

Overall Resume and Summary of Discussion if Discussed

Essentially unedited critiques

Budget recommendations

Administrative Notes

Available in Commons only to Principal Investigator

42

Example 1 – Phase II projectPROTECTION OF HUMAN SUBJECTS UNACCEPTABLE

RESUME AND SUMMARY OF DISCUSSION: The proposed project is intended to provide web-based education to and collaboration between parents and teachers of children with ADHD. The proposed program addresses a timely and important problem and if effective, this product would be a welcomed tool. The investigative team is exceptionally well qualified, Phase I successfully demonstrated the program’s feasibility and Phase II is well poised to refine the prototype. During discussion, however, the Committee noted several issues that limit the potential of the Phase II project. For example, the proposed assessment of the program’s effectiveness does not rely on parents and teachers of students with ADHD, and it does not assess the actual collaborative experiences of parent-teacher dyads that use it. Another concern centers on the plan to provide parents with clinical tools without proper guidance from mental health professionals. …There is also some question about the feasibility of creating a product that is relevant and useful for the full range of experience with ADHD that characterizes the target audience.

43

Example – Fast Track projectRESUME AND SUMMARY OF DISCUSSION: The proposed project develops a brief screen for depression in antenatal and postpartum women. Although it is unclear how feasible it is to obtain a reliable screening tool that is shorter than the 10-item Edinburgh Postpartum Depression Scale, if the project is successful the product would provide an important clinical tool. The project is especially significant because there is not an adequate scale to assess depressive symptoms among pregnant women who have physical symptoms. Using item response theory to develop the screener is innovative and will potentially result in a highly marketable product. The application outlines a strong commercialization plan. There are, however, some aspects of the proposed methodology that are questionable and there is concern that all of the members of the expert panel are directly involved in the project. Also Phase II lacks a gold standard assessment of depression for comparison with the final product. … These issues notwithstanding, the proposed product is innovative and, if successful, would be a significant contribution to the field.

44

ADVISORY COUNCIL/BOARD CONSIDERATION

• Second required step in dual peer review process

• Most applications considered in en bloc action; a few are specifically discussed (budget change, deferral)

45

QUESTIONS