Evidence-Based Funding: Thoughts About Extramural...
Transcript of Evidence-Based Funding: Thoughts About Extramural...
![Page 1: Evidence-Based Funding: Thoughts About Extramural Researchsites.nationalacademies.org/cs/groups/pgasite/documents/... · 2020-04-14 · Evidence-Based Funding: Thoughts About Extramural](https://reader033.fdocuments.net/reader033/viewer/2022060306/5f09a3707e708231d427cbc6/html5/thumbnails/1.jpg)
Evidence-Based Funding: Thoughts About Extramural Research
Michael S Lauer, MD Deputy Director for Extramural Research
National Institutes of Health
Plenary: Meeting of the Federal Demonstration Partnership (FDP) Thursday, May 5, 2016
Hyatt Regency Capitol Hill, 400 NJ Avenue, Washington, DC Disclosures: None
1
![Page 2: Evidence-Based Funding: Thoughts About Extramural Researchsites.nationalacademies.org/cs/groups/pgasite/documents/... · 2020-04-14 · Evidence-Based Funding: Thoughts About Extramural](https://reader033.fdocuments.net/reader033/viewer/2022060306/5f09a3707e708231d427cbc6/html5/thumbnails/2.jpg)
A Problem?
2 Shapiro D, Vrana K. PNAS 2015;112:9496-7
![Page 3: Evidence-Based Funding: Thoughts About Extramural Researchsites.nationalacademies.org/cs/groups/pgasite/documents/... · 2020-04-14 · Evidence-Based Funding: Thoughts About Extramural](https://reader033.fdocuments.net/reader033/viewer/2022060306/5f09a3707e708231d427cbc6/html5/thumbnails/3.jpg)
Mission Unchanged
3
“… same Mission and Purpose as we did in 1971 – to deliver People where they want to go, safely, ontime, with their bag, and with a smile ... With low fares and no hidden fees.” Gary Kelly CEO
http://swamedia.com/magazine
![Page 4: Evidence-Based Funding: Thoughts About Extramural Researchsites.nationalacademies.org/cs/groups/pgasite/documents/... · 2020-04-14 · Evidence-Based Funding: Thoughts About Extramural](https://reader033.fdocuments.net/reader033/viewer/2022060306/5f09a3707e708231d427cbc6/html5/thumbnails/4.jpg)
Quantitative Meaning!
• Where they want to go: % non-stops • Safely: 0% major incidents • On time: % of flights on time • With their bag: % of checked bags lost • With a smile: rate of complaints • Low fares: comparable measure
4
![Page 5: Evidence-Based Funding: Thoughts About Extramural Researchsites.nationalacademies.org/cs/groups/pgasite/documents/... · 2020-04-14 · Evidence-Based Funding: Thoughts About Extramural](https://reader033.fdocuments.net/reader033/viewer/2022060306/5f09a3707e708231d427cbc6/html5/thumbnails/5.jpg)
What Should Our Metrics Be?
5 Shapiro D, Vrana K. PNAS 2015;112:9496-7
![Page 6: Evidence-Based Funding: Thoughts About Extramural Researchsites.nationalacademies.org/cs/groups/pgasite/documents/... · 2020-04-14 · Evidence-Based Funding: Thoughts About Extramural](https://reader033.fdocuments.net/reader033/viewer/2022060306/5f09a3707e708231d427cbc6/html5/thumbnails/6.jpg)
Better Finish Line
6 Moses H et al. JAMA 2015;313:174-189
![Page 7: Evidence-Based Funding: Thoughts About Extramural Researchsites.nationalacademies.org/cs/groups/pgasite/documents/... · 2020-04-14 · Evidence-Based Funding: Thoughts About Extramural](https://reader033.fdocuments.net/reader033/viewer/2022060306/5f09a3707e708231d427cbc6/html5/thumbnails/7.jpg)
Funding
7
Moses H et al. JAMA 2015;313:174-189
![Page 8: Evidence-Based Funding: Thoughts About Extramural Researchsites.nationalacademies.org/cs/groups/pgasite/documents/... · 2020-04-14 · Evidence-Based Funding: Thoughts About Extramural](https://reader033.fdocuments.net/reader033/viewer/2022060306/5f09a3707e708231d427cbc6/html5/thumbnails/8.jpg)
Stagnant …
8 Moses H et al. JAMA 2015;313:174-189
Annual Growth 2004 – 2012: Pharma -0.6% Device +6.2% Biotech +4.6% NIH -1.8%
![Page 9: Evidence-Based Funding: Thoughts About Extramural Researchsites.nationalacademies.org/cs/groups/pgasite/documents/... · 2020-04-14 · Evidence-Based Funding: Thoughts About Extramural](https://reader033.fdocuments.net/reader033/viewer/2022060306/5f09a3707e708231d427cbc6/html5/thumbnails/9.jpg)
Another Perspective …
Press WH. Science 2013;342:817- 22
![Page 10: Evidence-Based Funding: Thoughts About Extramural Researchsites.nationalacademies.org/cs/groups/pgasite/documents/... · 2020-04-14 · Evidence-Based Funding: Thoughts About Extramural](https://reader033.fdocuments.net/reader033/viewer/2022060306/5f09a3707e708231d427cbc6/html5/thumbnails/10.jpg)
Trends that Matter …
Van Noorden RV. Nature 2014;505:144-5
![Page 11: Evidence-Based Funding: Thoughts About Extramural Researchsites.nationalacademies.org/cs/groups/pgasite/documents/... · 2020-04-14 · Evidence-Based Funding: Thoughts About Extramural](https://reader033.fdocuments.net/reader033/viewer/2022060306/5f09a3707e708231d427cbc6/html5/thumbnails/11.jpg)
Perhaps the Most Important Trend …
Press WH. Science 2013;342:817- 22
![Page 12: Evidence-Based Funding: Thoughts About Extramural Researchsites.nationalacademies.org/cs/groups/pgasite/documents/... · 2020-04-14 · Evidence-Based Funding: Thoughts About Extramural](https://reader033.fdocuments.net/reader033/viewer/2022060306/5f09a3707e708231d427cbc6/html5/thumbnails/12.jpg)
Workforce
12
Moses H et al. JAMA 2015;313:174-189
![Page 13: Evidence-Based Funding: Thoughts About Extramural Researchsites.nationalacademies.org/cs/groups/pgasite/documents/... · 2020-04-14 · Evidence-Based Funding: Thoughts About Extramural](https://reader033.fdocuments.net/reader033/viewer/2022060306/5f09a3707e708231d427cbc6/html5/thumbnails/13.jpg)
Distribution of funding Size of workforce
Are trainees students or workforce?
Optimizing education
Maximizing chances for discoveries
Reproducibility
Scientific rewards system
Diversifying Research
Promoting creativity and innovation
Stability and sustainability
Post-doc. holding tank
Early career investigators
Peer review
Academic business model
Efficiency
Thanks to Jon Lorsch
![Page 14: Evidence-Based Funding: Thoughts About Extramural Researchsites.nationalacademies.org/cs/groups/pgasite/documents/... · 2020-04-14 · Evidence-Based Funding: Thoughts About Extramural](https://reader033.fdocuments.net/reader033/viewer/2022060306/5f09a3707e708231d427cbc6/html5/thumbnails/14.jpg)
A Different Metric
14
Changing our funding metric “A question that at first glance may seem trivial but is, I believe, a significant one is whether our key metric for how… we invest in … research should be the number of grants we award or the number of investigators we support.”
Lorsch JR. Mol Biol Cell 2015;26:1578-82
![Page 15: Evidence-Based Funding: Thoughts About Extramural Researchsites.nationalacademies.org/cs/groups/pgasite/documents/... · 2020-04-14 · Evidence-Based Funding: Thoughts About Extramural](https://reader033.fdocuments.net/reader033/viewer/2022060306/5f09a3707e708231d427cbc6/html5/thumbnails/15.jpg)
15
Thanks to OER DPEA and SARB
![Page 16: Evidence-Based Funding: Thoughts About Extramural Researchsites.nationalacademies.org/cs/groups/pgasite/documents/... · 2020-04-14 · Evidence-Based Funding: Thoughts About Extramural](https://reader033.fdocuments.net/reader033/viewer/2022060306/5f09a3707e708231d427cbc6/html5/thumbnails/16.jpg)
16
Thanks to OER DPEA and SARB
![Page 17: Evidence-Based Funding: Thoughts About Extramural Researchsites.nationalacademies.org/cs/groups/pgasite/documents/... · 2020-04-14 · Evidence-Based Funding: Thoughts About Extramural](https://reader033.fdocuments.net/reader033/viewer/2022060306/5f09a3707e708231d427cbc6/html5/thumbnails/17.jpg)
17
Thanks to OER DPEA and SARB
![Page 18: Evidence-Based Funding: Thoughts About Extramural Researchsites.nationalacademies.org/cs/groups/pgasite/documents/... · 2020-04-14 · Evidence-Based Funding: Thoughts About Extramural](https://reader033.fdocuments.net/reader033/viewer/2022060306/5f09a3707e708231d427cbc6/html5/thumbnails/18.jpg)
18
Thanks to OER DPEA and SARB
![Page 19: Evidence-Based Funding: Thoughts About Extramural Researchsites.nationalacademies.org/cs/groups/pgasite/documents/... · 2020-04-14 · Evidence-Based Funding: Thoughts About Extramural](https://reader033.fdocuments.net/reader033/viewer/2022060306/5f09a3707e708231d427cbc6/html5/thumbnails/19.jpg)
A Difficult Conversation …
19
Alberts B et al. PNAS. 2014;111:5773-7
“… unsustainable hypercompetitive system ... a recipe for long-term decline ... rethink some fundamental features of the US biomedical research ecosystem...
![Page 20: Evidence-Based Funding: Thoughts About Extramural Researchsites.nationalacademies.org/cs/groups/pgasite/documents/... · 2020-04-14 · Evidence-Based Funding: Thoughts About Extramural](https://reader033.fdocuments.net/reader033/viewer/2022060306/5f09a3707e708231d427cbc6/html5/thumbnails/20.jpg)
20
![Page 21: Evidence-Based Funding: Thoughts About Extramural Researchsites.nationalacademies.org/cs/groups/pgasite/documents/... · 2020-04-14 · Evidence-Based Funding: Thoughts About Extramural](https://reader033.fdocuments.net/reader033/viewer/2022060306/5f09a3707e708231d427cbc6/html5/thumbnails/21.jpg)
Publication and Citations?
21 Shapiro D, Vrana K. PNAS 2015;112:9496-7
![Page 22: Evidence-Based Funding: Thoughts About Extramural Researchsites.nationalacademies.org/cs/groups/pgasite/documents/... · 2020-04-14 · Evidence-Based Funding: Thoughts About Extramural](https://reader033.fdocuments.net/reader033/viewer/2022060306/5f09a3707e708231d427cbc6/html5/thumbnails/22.jpg)
So How Do We Measure Research Impact?
• P = productivity – Publish trial results – Highly-cited papers: field-normalized
• Q = quality • R = replication • S = sharing • T = translation
22 Ioannidis JP, Khoury MJ. JAMA 2014 (June 9)
![Page 23: Evidence-Based Funding: Thoughts About Extramural Researchsites.nationalacademies.org/cs/groups/pgasite/documents/... · 2020-04-14 · Evidence-Based Funding: Thoughts About Extramural](https://reader033.fdocuments.net/reader033/viewer/2022060306/5f09a3707e708231d427cbc6/html5/thumbnails/23.jpg)
Cautions with Bibliometrics
23
Hicks D, Wouters P et al. Nature 2015;520:429-31 Bornmann L, Marx W. EMBO Reports 2013;14:226-30
![Page 24: Evidence-Based Funding: Thoughts About Extramural Researchsites.nationalacademies.org/cs/groups/pgasite/documents/... · 2020-04-14 · Evidence-Based Funding: Thoughts About Extramural](https://reader033.fdocuments.net/reader033/viewer/2022060306/5f09a3707e708231d427cbc6/html5/thumbnails/24.jpg)
Grant Sample
• NIH Type 1 RPG funded 1985 - 2012 • Data from IMPAC II, SPIRES
– Grants = 135,315 – Papers = 793,409 – Scientists = 65,858 – Organizations = 2549
24
![Page 25: Evidence-Based Funding: Thoughts About Extramural Researchsites.nationalacademies.org/cs/groups/pgasite/documents/... · 2020-04-14 · Evidence-Based Funding: Thoughts About Extramural](https://reader033.fdocuments.net/reader033/viewer/2022060306/5f09a3707e708231d427cbc6/html5/thumbnails/25.jpg)
Bibliometric Outcomes
• Data from Thomson Reuters (95% match) • Each paper binned by:
– Topic (empiric, one of 252) – Year of publication; type (research, review)
• Each paper percentiled (0-100%) • Sum to calculate grant citation impact
25
Hicks D, Wouters P et al. Nature 2015;520:429-31 Bornmann L, Marx W. EMBO Reports 2013;14:226-30
![Page 26: Evidence-Based Funding: Thoughts About Extramural Researchsites.nationalacademies.org/cs/groups/pgasite/documents/... · 2020-04-14 · Evidence-Based Funding: Thoughts About Extramural](https://reader033.fdocuments.net/reader033/viewer/2022060306/5f09a3707e708231d427cbc6/html5/thumbnails/26.jpg)
Quick Review of Box Plots
26
http://patilv.com/img/2013-11-7-Replication-of-few-graphs-charts-in-base-R-ggplot2-and-rCharts-part-2-ggplot2/unnamed-chunk-34.png
Median
75th percentile
25th percentile
![Page 27: Evidence-Based Funding: Thoughts About Extramural Researchsites.nationalacademies.org/cs/groups/pgasite/documents/... · 2020-04-14 · Evidence-Based Funding: Thoughts About Extramural](https://reader033.fdocuments.net/reader033/viewer/2022060306/5f09a3707e708231d427cbc6/html5/thumbnails/27.jpg)
27
![Page 28: Evidence-Based Funding: Thoughts About Extramural Researchsites.nationalacademies.org/cs/groups/pgasite/documents/... · 2020-04-14 · Evidence-Based Funding: Thoughts About Extramural](https://reader033.fdocuments.net/reader033/viewer/2022060306/5f09a3707e708231d427cbc6/html5/thumbnails/28.jpg)
28
![Page 29: Evidence-Based Funding: Thoughts About Extramural Researchsites.nationalacademies.org/cs/groups/pgasite/documents/... · 2020-04-14 · Evidence-Based Funding: Thoughts About Extramural](https://reader033.fdocuments.net/reader033/viewer/2022060306/5f09a3707e708231d427cbc6/html5/thumbnails/29.jpg)
29
![Page 30: Evidence-Based Funding: Thoughts About Extramural Researchsites.nationalacademies.org/cs/groups/pgasite/documents/... · 2020-04-14 · Evidence-Based Funding: Thoughts About Extramural](https://reader033.fdocuments.net/reader033/viewer/2022060306/5f09a3707e708231d427cbc6/html5/thumbnails/30.jpg)
30
![Page 31: Evidence-Based Funding: Thoughts About Extramural Researchsites.nationalacademies.org/cs/groups/pgasite/documents/... · 2020-04-14 · Evidence-Based Funding: Thoughts About Extramural](https://reader033.fdocuments.net/reader033/viewer/2022060306/5f09a3707e708231d427cbc6/html5/thumbnails/31.jpg)
31
![Page 32: Evidence-Based Funding: Thoughts About Extramural Researchsites.nationalacademies.org/cs/groups/pgasite/documents/... · 2020-04-14 · Evidence-Based Funding: Thoughts About Extramural](https://reader033.fdocuments.net/reader033/viewer/2022060306/5f09a3707e708231d427cbc6/html5/thumbnails/32.jpg)
“Payoffs from research follow a power law, with big, near-unlimited upside but limited downside. Since the winner will have an explosive payoff, the right approach requires a certain style of blind funding … [of] as large a number of projects as possible...” Nassim Nicholas Taleb
What the Power Law Means …
Taleb N 2012 Random House
![Page 33: Evidence-Based Funding: Thoughts About Extramural Researchsites.nationalacademies.org/cs/groups/pgasite/documents/... · 2020-04-14 · Evidence-Based Funding: Thoughts About Extramural](https://reader033.fdocuments.net/reader033/viewer/2022060306/5f09a3707e708231d427cbc6/html5/thumbnails/33.jpg)
The Power Law is Everywhere
33
“Why do social scientists care? One implication of the [power] law is that there are many more extreme events than would occur if the distribution were Gaussian; for example, if the distribution were the result of a series of events of fixed probability, like flipping coins or rolling dice.”
Gabaix X. J Econ Persp 2016;30:185-206
![Page 34: Evidence-Based Funding: Thoughts About Extramural Researchsites.nationalacademies.org/cs/groups/pgasite/documents/... · 2020-04-14 · Evidence-Based Funding: Thoughts About Extramural](https://reader033.fdocuments.net/reader033/viewer/2022060306/5f09a3707e708231d427cbc6/html5/thumbnails/34.jpg)
An NIH Approach
34
![Page 35: Evidence-Based Funding: Thoughts About Extramural Researchsites.nationalacademies.org/cs/groups/pgasite/documents/... · 2020-04-14 · Evidence-Based Funding: Thoughts About Extramural](https://reader033.fdocuments.net/reader033/viewer/2022060306/5f09a3707e708231d427cbc6/html5/thumbnails/35.jpg)
Programs, not Individual Projects
• Stability of funding • Flexibility – “follow your nose” • Distribution of funding • Less time writing grant applications • Less time reviewing grant applications
35
https://publications.nigms.nih.gov/presentations/MIRA-Reissue-RFA-GM-17-002-Webinar-Revised.pdf
![Page 36: Evidence-Based Funding: Thoughts About Extramural Researchsites.nationalacademies.org/cs/groups/pgasite/documents/... · 2020-04-14 · Evidence-Based Funding: Thoughts About Extramural](https://reader033.fdocuments.net/reader033/viewer/2022060306/5f09a3707e708231d427cbc6/html5/thumbnails/36.jpg)
With an Important Variation
36
![Page 37: Evidence-Based Funding: Thoughts About Extramural Researchsites.nationalacademies.org/cs/groups/pgasite/documents/... · 2020-04-14 · Evidence-Based Funding: Thoughts About Extramural](https://reader033.fdocuments.net/reader033/viewer/2022060306/5f09a3707e708231d427cbc6/html5/thumbnails/37.jpg)
Another Finish Line?
37 Shapiro D, Vrana K. PNAS 2015;112:9496-7
![Page 38: Evidence-Based Funding: Thoughts About Extramural Researchsites.nationalacademies.org/cs/groups/pgasite/documents/... · 2020-04-14 · Evidence-Based Funding: Thoughts About Extramural](https://reader033.fdocuments.net/reader033/viewer/2022060306/5f09a3707e708231d427cbc6/html5/thumbnails/38.jpg)
38
Ivacaftor
FDA trials
Documents / Publications
Scientists (N=2587) Institutions (N=2516)
Sanders Williams R et al. Cell 2015;163:21-23
![Page 39: Evidence-Based Funding: Thoughts About Extramural Researchsites.nationalacademies.org/cs/groups/pgasite/documents/... · 2020-04-14 · Evidence-Based Funding: Thoughts About Extramural](https://reader033.fdocuments.net/reader033/viewer/2022060306/5f09a3707e708231d427cbc6/html5/thumbnails/39.jpg)
Cited References
NIH Supported Institutions NIH
Grants
Foundational Publications
Clinical Trials
Repatha
Telling a Story with Big Data
Thanks Jim Onken, Brian Haugan, George Chacko, Shixin Jiang, Samet Keserci, Alex Pico, and Lindsay Wan
![Page 40: Evidence-Based Funding: Thoughts About Extramural Researchsites.nationalacademies.org/cs/groups/pgasite/documents/... · 2020-04-14 · Evidence-Based Funding: Thoughts About Extramural](https://reader033.fdocuments.net/reader033/viewer/2022060306/5f09a3707e708231d427cbc6/html5/thumbnails/40.jpg)
NIH Objectives
40 http://www.nih.gov/sites/default/files/about-nih/strategic-plan-fy2016-2020-508.pdf
• Science of science • Outputs, outcomes • Workforce analyses • Review peer review • Rigor, reproducibility • Administrative burden • Risk management
![Page 41: Evidence-Based Funding: Thoughts About Extramural Researchsites.nationalacademies.org/cs/groups/pgasite/documents/... · 2020-04-14 · Evidence-Based Funding: Thoughts About Extramural](https://reader033.fdocuments.net/reader033/viewer/2022060306/5f09a3707e708231d427cbc6/html5/thumbnails/41.jpg)
Others are Singing This Tune …
41
Azoulay P. Nature 2012;484:31-32 Ioannidis J. Nature 2011; 477:529-31
“It is a scandal that billions of dollars are spent on research without knowing the best way to distribute that money.” John Ioannidis
![Page 42: Evidence-Based Funding: Thoughts About Extramural Researchsites.nationalacademies.org/cs/groups/pgasite/documents/... · 2020-04-14 · Evidence-Based Funding: Thoughts About Extramural](https://reader033.fdocuments.net/reader033/viewer/2022060306/5f09a3707e708231d427cbc6/html5/thumbnails/42.jpg)
Looking Forward to Open Dialogue …
42 http://nexus.od.nih.gov/all/category/open-mike/