Richard Nakamura. PhD

CSR Advisory CouncilMay 2014

Strategy for quality measurement

In most production systems, there is a speed, cost, quality

trade-off, but this is not a zero-sum game.

We have just discussed speed and to some extent, cost.

EffectivenessReview quality

For Problem A, we have focused on:

Attracting:…the best reviewers …the best Chairs …the best SROs

After we put together the best review committees, they must work in the best platform/format to determine merit of applications

Best reviewers -…recruit mostly senior scientists …funded awardees of NIH…w/ strong publications…positive review experience …respect from other reviewers…and diversity that aids the SRG

To get the best reviewers, we offer -…continuous submission…new forms of attendance…meetings on West coast

But -…we take away coffee…we make travel less convenient…we issue 1099 forms for all payments

Today, NIH wants to know how to measure quality so that we can systematically improve it.

To the extent that we have a standard, it has been to examine the results of grants ten years after award as evaluated by senior scientists. This has been done for the Pioneer awards (for 7 years) but provides a very slow feedback loop. Can we do better?

We would like to have measures available more immediately than 10 years after award that predict the longer outcome.

How do we get there?To begin with we need to have estimates of the reliability of review scoring (Rene Etcheberrigaray), so we can power our other measures of quality and interventions correctly.

We need evidence that our scoring approaches are not inappropriately biased – Monica Basco.

In measuring quality, CSR takes seriously the opinions of our scientific reviewers:

Surveys – CSR quick feedback (Mary Ann Guadagno)

Committee ranking within IRGS (Problem B and last Council)

Is the application ranking system working? (Amy Rubinstein)

Do our committees favor conformity? (Seymour Garte)

What other approaches could help us evaluate quality? (actively pursued by other groups and not covered today)

Bibliometrics (DPCPSI and CSR):Citations at individual, group levelsRelative citation rating; h-indexConcerns about citation manipulation

Review committee assignments and coverage

Network analysis - Application and publication analysis of applicants and reviewers

Creation of computer assisted assignment

AARRBBBPBCMBBDCNBSTCBCVRSDKUSEMNRETTNGGGHDMIDMIFCNIMMMDCNMOSSOBTOTCPSERPHBSBIBVHOther

A S

yste

m L

evel

Rep

rese

ntat

ion

of

Sci

entif

ic In

tere

sts

Biological Chemistry & Macromolecular Biophysics

Infectious Diseases and Microbiology

Questions? Comments?

CSRDirector@csr.nih.gov