PCORI’s Merit Review Process

Mari  Kimura,  PhD  Merit  Review  Officer    February  20,  2015    

Mari Kimura, PhD Merit Review Officer

Welcome!

2

Session Objectives

•   Introduce  PCORI  Merit  Reviewers  –  Who  are  the  reviewers?  

•   Outline  the  Merit  Review  process  –  What  happens  to  applicaCons  aDer  submission?  –  How  do  the  reviewers  evaluate  applicaCons?  

 

3

Goal of PCORI Merit Review

4

To identify applications that have the strongest potential to improve patient outcomes

Building an Inclusive Merit Review

•  Panels  include  scienCsts,  paCents,  and  other  stakeholders  to  bring  diverse  perspecCves  to  the  review  process.    

•  Each  applicaCon  is  assigned  to  2  scienCsts,  1  paCent,  and  1  other  stakeholder.  

•  Chair  facilitates  discussion  and  promotes  a  culture  of  mutual  respect  and  understanding  among  reviewer  types  

5

Patients Other Stakeholders

Scientists

Reviewer Eligibility

•   ScienCsts  –   Advanced  degree  in  health  or  

research-­‐related  field    –   Relevant  peer-­‐reviewed  

publicaCons  –   AcCve  research  program  in  a  

relevant  field  of  study    •   Chairs  

–   NaConal  reputaCon  in  research  area  

–   Demonstrated  leadership  experience  

6

Reviewer Eligibility

•   Stakeholders  –   Contribute  a  unique  healthcare  

system  perspecCve  •   PaCents  

–   PaCent  advocates  –   Caregivers  

•   Excluded  –   Board  of  Governors  –   SelecCon  CommiWee  members  –   Applicants      

7

Where Do Our Reviewers Come From?

8

Merit Reviewers by Role

9

N=3897  

Stakeholder  32%  

Pa<ent  12%  

Scien<fic  56%  

Diverse Stakeholder Roles in Reviewer Pool

10

Clinician 33%

Hospital or Health System

11% Industry

5% Payer

2%

Policy Maker 4%

Researcher 35%

Training Institution

5%

Unsure 5%

11

Responsiveness Review

12

•   LeWers  of  intent  (LOIs)  are  reviewed  (by  PCORI  or  external  reviewers)  based  on  criteria  detailed  in  each  PCORI  funding  announcement  (PFA).  

•   AddiConal  screening  for  –   ComparaCve  effecCveness  research  –   Exclusion  of  cost-­‐effecCveness  analysis  –   ProgrammaCc  concerns  

•   Only  responsive  LOIs  will  be  invited  to  submit  a  full  applicaCon.  

Comparative Effectiveness Research

Exclusion of Cost-Effectiveness Analysis

Programmatic Fit

Recruiting Reviewers

•   Targeted  recruitment  of  reviewers  and  panel  Chair  following  based  on  topic  areas  of  accepted  LOIs  

•   Reviewers  invited  based  on  experCse/perspecCve  

13

14

Application Assignments

15

ApplicaCon  assignments  based  on  

ExperCse   ConsideraCon  of  COI  

About  6  applicaCons  per  reviewer  

All  panel  reviewers  have  training  

Web  Based  Training  

Program-­‐led  Webinars  

Approximately  4  weeks  to  review  assigned  applicaCons  

PaCent  and  Other  Stakeholder  reviewers  receive  addiConal  training  

Mentor  program  supplements  

training  

AddiConal  training  aids  such  as  

Methodology  101  

Mentor Program

16

Complete  wriWen  criCques  

that  are  informaCve  for  applicants  and  program  staff  

ParCcipate  effecCvely  at  the  panel  meeCng  

Enjoy  parCcipaCng  in  the  PCORI  Merit  Review  process  

Provides  paCent  and  

other  stakeholder  reviewers  the  support  they  

need  to  

17

Critique Review by MROs and Mentors

•   All  reviewers  write  a  criCque  for  each  of  their  assigned  applicaCons  and  provide  criterion  scores  and  overall  scores  

•   All  reviewers  will  receive  feedback  on  wriWen  criCques  from    Mentors  and  Merit  Review  Officers  who  review  criCques    as  panel  members  complete  them  –   Clarifies  goals  of  PFA  and  content  of  criCques  –   Helps  reviewers  express  their  unique    perspecCve  in  a  manner  that  is  acConable    by  the  applicant  

–   Ensures  fair  and  consistent  review    

18

Merit Review Criteria

19

Criterion  #1:  Impact  of  the  condiCon  on            the  health  of  individuals  and  populaCon    

Criterion  #2:  PotenCal  for  the  study  to  improve  healthcare  and  outcomes  

Criterion  #3:  Technical  merit  

Criterion  #4:  PaCent-­‐centeredness  

Criterion  #5:  PaCent  and  other  stakeholder  engagement    

Pa<ent  and  Other  

Stakeholder  Reviewers  

Scien<st  Reviewers  

 

   

 

   

   

We Require Patient-Centeredness and Patient and Other Stakeholder Engagement

Patient and Other Stakeholder Engagement

•   PaCents  are  partners  in  research,    not  just  “subjects”  

•   AcCve  and  meaningful  engagement  between  scienCsts,  paCents,  and  other  stakeholders  

•   Community,  paCent,  and  caregiver  involvement  already  in  existence  or  a  well-­‐thought  out  plan  throughout  the  project  

Patient-Centeredness •   Does  the  project  aim  to  answer  

quesCons  or  examine  outcomes  that  maWer  to  paCents  within  the  context  of  paCent  preferences?  

•   Research  quesCons  and  outcomes  should  reflect  what  is  important  to  paCents  and  caregivers  

20

Additional Considerations

21

Adherence  to  Methodology  Standards  

Human  Subjects  

ProtecCons  

Scoring Range

22

Range   Score   Descriptor   Characteris<cs  

High  

1   ExcepConal   ExcepConally  strong  with  essenCally  no  weaknesses  

2   Outstanding   Extremely  strong  with  negligible  weaknesses  

3   Excellent   Very  strong  with  only  some  minor  weaknesses  

Medium  

4   Very  Good   Strong  but  with  numerous  minor  weaknesses  

5   Good   Strong  but  with  at  least  one  moderate  weakness  

6   SaCsfactory   Some  strengths  but  also  some  moderate  weakness  

Low  

7   Fair   Some  strengths  but  with  at  least  one  major  weakness  

8   Marginal   A  few  strengths  and  a  few  major  weaknesses  

9   Poor   Very  few  strengths  and  numerous  major  weaknesses  

A  score  of  1  indicates  an  excepConally  strong  applicaCon.    

A  score  of  9  indicates  an  applicaCon  with  serious  and  substanCve  weaknesses.  

23

Setting the Discussion Line

24

PCORI  program  staff  consider  numerical  scores  

ApplicaCons  of  parCcular  interest  to  programs  can  be  included  in  discussion  

ProporCon  of  applicaCons  discussed  depends  on  scores  and  varies  among  PFAs  

25

Merit Review In-Person Meeting

26

Chair Introduces Application

Reviewer 1: Scientist 1

Reviewer 2: Patient

Reviewer 3: Stakeholder

Reviewer 4: Scientist 2

Descrip<on

Chair  briefly  introduces  applicaCon

4  lead  reviewers  present  score  and  highlights  of  their  criCques General  panel  discussion  facilitated  by  Chair Chair  summarizes  panel  discussion  of  applicaCon   All  panel  reviewers  score  applicaCon

Discussion Components

•   The  five  Merit  Review  criteria  •   ProgrammaCc  goals  as  defined  in  the  PFA  

How  each  applica<on  addresses  

•  Methodological  rigor  in  a  project  leads  to  trustworthy,  paCent-­‐centered  informaCon  

Emphasis  on  importance  of  rigorous  science  

•   Responsiveness  to  comments  from  original  review  are  considered  but  not  a  guarantee  of  funding  

Resubmissions  reviewed  as  standalone  applica<on  

27

Strategies for Preparing a Successful Funding Application

28

Your  applicaCon  should  tell  the  story  of  why  the  proposed  study  is  important  to  all  reviewers  

Don’t  assume  reviewers  know  something  about  your  project  that  you  don’t  address  

Be  clear  on  what  is  to  be  gained  if  your  study  is  funded  

28

NOTE   All of the guidance provided to reviewers is available at PCORI.org under Reviewer Resources.

29

30

31

Summary Statements

•  All  applicants  receive  a  summary  statement  at  the  end  of  the  review  cycle.  

•   Preliminary  reviewer  criCques  

•   Notes  from  applicaCon  discussion  

•   Final  panel  average  overall  score  

If  the  applica<on  is  discussed,  summary  statement  includes:  

•   Preliminary  reviewer  criCques  

If  the  applica<on  is  not  discussed,  summary  statement  includes:  

32

Become a Reviewer

•   PCORI  review  panels  include  scienCsts,  paCents,  and  other  stakeholders  to  bring  diverse  perspecCves  to  the  review  process.  

•   We  conCnue  to  welcome  applicaCons  to  become  a  reviewer  in  all  categories.  

33

How to Apply to be a PCORI Reviewer

Visit: www.pcori.org/content/become-reviewer

•   Qualifications

•   Reviewer Expectations

•   Compensation

•   Training Materials

34

Questions?

Thank You! Mari Kimura, PhD Merit Review Officer