Using Metrics to inform the Return on Investment (ROI): A ... · profiles, protein and RNA...
Transcript of Using Metrics to inform the Return on Investment (ROI): A ... · profiles, protein and RNA...
Using Metrics to inform the Return on Investment (ROI): A Public Funder’s Perspective
Sean Coady, Division of Cardiovascular Sciences, National Heart, Lung, and Blood Institute
NIH Virtual Workshop on Data Metrics February 19, 2020
2
NHLBI Data/Materials Sharing
database of Genotypes and Phenotypes (dbGaP), GWAS, microRNA, gene expression profiling
Trans-Omics for Precision Medicine (TOPMed); WGS, metabolic profiles, protein and RNA expression, DNA methylation
BioData Catalyst Storage, Toolspace, Access and analytics for big data Empowerment
BioLogic specimen & data repositories INformation Coordinating Center (BioLINCC); biospecimens & phenotypic data
Clinical TrialsObservational Studies
3
Background – Data Repository
1999 NHLBI IRB approves data repository protocol. Website opens in 2000 (NIH IRB # 12-H-N198)
Purpose:In order to take full advantage of NHLBI supported clinical trials and epidemiologic studies and maximize their research value, data should be made available, under appropriate terms and conditions, to the largest possible number of qualified investigators in a timely manner.
Background BioLINCC
BioLINCC established in 2009 to coordinate activities of Data Repository and Biorepository
4
5
Current controlled access portfolio: 50 Observational studies (778,732 participants), 157 trials (460,816 participants)
0 2 4 6 8 10 12 14 16 18 20 22 24 26 28
Cardiovascular CTTransplant/Transfusion CT
Primary/Sec. Prevention CTAsthma CT
Other CTOther OS
Heart Failure CTDisorder Population OS
Other Lung CTEmergency Medicine CT
General Population OSRespiratory Distress CT
Race/Ethnic Specific OSSickle Cell CT
Number of Studies
Data Repository Metrics
Utilization Metrics
Outcome Metrics
Effort Metrics Number of interactions and time to complete data access request Support encounters Monitoring
6
Utilization Metrics: Completed data access requests
0
50
100
150
200
250
300
350
400
2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016 2017 2018 2019
Fulfi
lled
Dat
a Ac
cess
Req
uest
s
7
Utilization Metrics: Cumulative data access requests for trial and observational study data (thru, 2019)
8
Utilization Metrics: Observed / Expected Utilization by study type
9
0 1 2 3 4 5 6 7 8 9 10
General Population OSRace/Ethnic Specific OSRespiratory Distress CT
Primary/Sec. Prevention CTHeart Failure CT
Disorder Population OSCardiovascular CT
Other OSOther CT
Other Lung CTEmergency Medicine CT
Sickle Cell CTAsthma CT
Transplant/Transfusion CT
Observed Data Requests (% of Total)/ Expected Data Requests (% of Portfolio)
10
Utilization Metrics: Trends in Utilization by Study Type
0.0 2.0 4.0 6.0 8.0 10.0 12.0 14.0 16.0 18.0 20.0 22.0 24.0
General Population OSRespiratory Distress CT
Heart Failure CTRace or Ethnicity Population OS
Primary/Sec. Prevention CTCardiovascular CT
Other OSDisorder Population OS
Emergency Medicine CTOther CT
Sickle Cell CTOther Lung CT
Transplant/Transfusion CTAsthma CT
Average number of data access requests per study per year
2013-2015 2016-2019
Utilization Metrics: Primary reason for data access request
11
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
2008 2009 2010 2011 2012 2013 2014 2015 2016 2017 2018 2019
1. New Question 2. Meta Analysis 3. Statistical Methods
4. Clinical Trial Methods 5. Comparison Group 6. Clinical Prediction, Risk Prediction
7. Other (Pilot, Simulation)
Utilization Metrics: Time from availability of study to first access request
CT: 60.8%OS: 74.8%
CT: 83.5%OS: 91.6%
CT:. Median 270 days.OS: Median 205 daysLog-rank p=0.296
12
Outcome Metrics: Publications
0
20
40
60
80
100
120
140
160
2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016 2017 2018 2019
Cou
nt o
f pub
licat
ions
Number of publications per calendar year
13
Outcome Metrics: Time to “Incident” publication by type of study (through 2019)
14
Log-rank p=0.858
1 YearCT: 7.3%OS: 2.1%
3 YearCT: 22.0%OS: 25.4%
5 YearCT: 41.5%OS: 46.1%
1179 publications596 Included data from 1 or more trials
69 Trials 1+ publications
506 investigators have published at least once. Mean number of publications per investigator publishing=3.0
15
Outcome Metrics: Citation Percentile (Top N%) publications using repository trial data, observational data and 10% random sample of all NHLBI supported articles (Articles thru May 2015)
Kruskal-Wallis Test p=0.333
Outcome Metrics: Workforce training
23% of completed data access requests indicate primary user has 0-5 years of research experience
QVR search of NIH grant applications found only two training applications (1 K01, 1 K99) mentioned BioLINCC
16
Messages from the metrics
Demand for secondary use of data from clinical strudiescontinues to steadily increase
Growing demand for data from clinical trials Metrics can suggest gap areas Data from repositories fulfill a range of purposes Unclear if publication rates are low Citation metrics suggest quality of publications utilizing repository
data are similar to publications supported directly by the Institute Need to better assess role of repositories in training new
investigators
17