Sharing, Reproducibility, Replication – AN NIH View
19
Sharing, Reproducibility, Replication – AN NIH View ACS National Meeting March 24, 2015 Philip E. Bourne, PhD, FAMCI Associate Director for Data Science, NIH Department of Health and Human Services With Thanks to Larry Tabak 1
-
Upload
philip-bourne -
Category
Education
-
view
886 -
download
2
Transcript of Sharing, Reproducibility, Replication – AN NIH View
Sharing, Reproducibility, Replication – AN NIH View
ACS National MeetingMarch 24, 2015
Philip E. Bourne, PhD, FAMCI
Associate Director for Data Science, NIHDepartment of Health and Human Services
With Thanks to Larry Tabak1
The Growing Challenge
Noted by research community; in multiple publications
Across research areas
Especially in preclinical research
2
Challenges to Ensuring Rigor and Transparency in Reporting Science: Additional Contributors
Insufficient Reporting
“P-Hacking”
Lack of Consideration of Sex as a Biological Variable
3
Challenges to Ensuring Rigor and Transparency in Reporting Science: Underlying Issues
4
Publish or perish! Grant support
Impact factor Innovation
Significance Novelty:
No negative dataPoor training
Incentives
5
Principles for Addressing the Underlying Issues
Raise community awareness
Enhance formal training
Protect the quality of funded and published research by adoption of more systematic review processes
Increase stability for investigators
6
Addressing Underlying Issues: Raise Community Awareness
Workshop in June 2014 with Journal Editors to identify common opportunity areas
Workshop in July 2014 with PhRMA to identify areas of common interest with industry
Obtained input on barriers to reproducibility re: research reagents
Meetings with professional societies and institutions
7
Addressing Underlying Issues:Raise Community Awareness
Over 130 journals endorsed the principles, which were broadly shared in November 2014 through editorials and other notifications
8http://nih.gov/about/endorsing-jounals.htm
Addressing Underlying Issues: Trans-NIH Pilots
9
Pilot Focus Types of Efforts Being Developed
Evaluation of scientific premise in grant applications
New Funding Opportunities with additional review criteria regarding scientific premise
Checklist and reporting guidelines Reviewer checklists regarding reporting standards and scientific rigor
Changes to biosketch Biosketch pilot with focus on accomplishments and not just publications
Approaches to reduce "perverse incentives” to publish
Exploring award options with a longer period of support for investigators
Supporting replication studies New Funding Opportunities for replication studies, and options to assess whether pre-clinical findings should be replicated
Training Developing materials on experimental design
Other efforts Use of Prize Challenges to encourage reproducibility of results, PubMed Commons
http://www.ncbi.nlm.nih.gov/pubmedcommons/
Addressing Underlying Issues:PubMed Commons
10
System allowing researchers to share opinions on publications indexed by PubMed
Issues Specific to Data Science
11
Elements of The Digital Enterprise
Communities Policies
Infrastructure
• Intersection:
• Sustainability
• Efficiency
• Collaboration
• Training
Policies: Now & Forthcoming
Data Sharing
Genomic data sharing announced
Data sharing plans on all research awards
Data sharing plan enforcement
Machine readable plan
Repository requirements to include grant
numbers
http://www.nih.gov/news/health/aug2014/od-27.htm
Policies - Forthcoming
Data Citation
Goal: legitimize data as a form of scholarship
Process:
Machine readable standard for data citation (done)
Endorsement of data citation for inclusion in NIH bib
sketch, grants, reports, etc.
Example formats for human readable data citations
Slowly work into NLM/NCBI workflow
dbGaP in the cloud (soon!)
BD2K
Center
BD2K
Center
BD2K
Center
BD2K
CenterBD2K
Center
BD2K
Center
DDICC
Software
Standards
Infrastructure - The
CommonsLabs
Labs
Labs
Labs
The Commons
Digital Objects (with UIDs)
Search(indexed metadata)
Computing Platform
The C
om
mons
Vivien Bonazzi
George Komatsoulis
The Commons: Compute
Platforms
The CommonsConceptual Framework
Public Cloud
Platforms
Super Computing (HPC) Platforms
OtherPlatforms
?
Google, AWS (Amazon)
Microsoft (Azure), IBM,
other?
In house compute
solutions
Private clouds, HPC
– Pharma
– The Broad
– Bionimbus
Traditionally low access
by NIH
The Commons:
Business Model
[George Komatsoulis]
