American Evaluation Association Annual Meeting, November 2011 Using Multiple Methods and Data...

American Evaluation Association Annual Meeting, November 2011

Using Multiple Methods and Data Sources to Analyze Complex Cancer Research

Portfolios

Joshua SchnellDiscovery Logic, A Thomson Reuters Business

Co-Authors: Elizabeth Hsu, Jim Corrigan (NIH), Sandeep Patel, Lauren Taffe, Duane Williams (Discovery Logic)


2

Data-driven portfolio management

The National Institutes of Health seeks to correlate the research and training it funds with subsequent public health impact.

Basic Research Findings

PublicationsCitations

Conference ProceedingsBiomarkers

Technology Development

InventionsPatents

Biomarkers

Clinical Development

Clinical trialsBiomarkers

Approved Products

FDA Approvals of Drugs,

Devices and Biologics

Public Health Impact

Population Results Patient

OutcomesBiomarkers


3

Robust, repeatable data linkages support portfolio analysis focused systems





InventionsPatents

Biomarkers



Approved Products





OutcomesBiomarkers

NIH’s electronic Scientific Portfolio Assistant (e-SPA) currently provides linkages to early stage portfolio outputs and impact.


4

Expanding the range of data-driven portfolio analysis

NCI’s Office of Science Planning and Assessment is conducting analyses of additional outputs such as cancer Biomarkers and FDA approved drugs.





InventionsPatents

Biomarkers



Approved Products





OutcomesBiomarkers


5

How has the NCI supported the development of cancer biomarkers? Background

• Feasibility study to link funding data to breast cancer biomarkers• Thomson Reuters’ Integrity platform includes Biomarkers

- High quality, manually curated data - Coverage of various cancer disease areas

• Linkage to NCI funded projects through supporting reference material- Research publications considered by the Integrity editorial team as

important for development of breast cancer biomarkers. - Research publications acknowledging NCI project support


6

How has the NCI supported the development of cancer biomarkers? Methodology

Patent

Indication

Biomarker

Acknowledgement of

NCI SupportPublication

Validity Status

Integrity Biomarkers Data

NIH Projects (IMPAC II)

924 Biomarkers

2,040 Biomarkers with different validity statuses for different breast cancer indications (e.g. in situ ductal carcinoma)


7

How has the NCI supported the development of cancer biomarkers? Findings

Emerging

Experimental

Early Studies in Humans

Late Studies in Humans

Recommended / Approved

0 100 200 300 400 500 600 700 800 900

Breast Cancer Biomarkers*NCI-funded Biomarkers / Indications Other

*Individual Biomarkers may be in multiple validity stages, depending on indication, and reference n=1,821. Total number of unique breast cancer Biomarkers = 924; Biomarkers supported by NCI n=456

SOURCES: Thomson Reuters Integrity, ScienceWire, NIH IMPAC II (May 2011)

32.4%

54.4%

32.0%

59.5%

66.7%


8

How has the NCI supported the development of cancer biomarkers? Findings, continued

Breast Cancer Biomarker Validity StatusSupporting NCI

Projects IdentifiedRecommended / Approved 92

Late Studies in Humans 208Early Studies in Humans 238

Experimental 239Emerging 198

TOTAL UNIQUE PROJECTS 456

Grant Mechanisms IncludedR grants = 241 (181 are R01s)U and N grants = 100P grants = 68Z grants = 10Training (F,K,L,T) = 35


9

How has the NCI supported the development of cancer biomarkers? Portfolio Implications

• Feasibility study has demonstrated:• The utility of linking NCI projects to biomarker outputs• The role of NCI support in breast cancer biomarker development

• New portfolio management questions can now be asked, such as:• What mechanisms are most effective at supporting the various

biomarker stage development? • What is the total funding investment made for biomarker

development? • What diseases lack biomarkers and should be considered for

targeted support?


10

How has the NCI supported the development of FDA approved drugs? Background

NCI Project[IMPACII]

Patent Acknowledgement of

NCI Support

NDA-Listed Patent[USPTO Patent]

Approved Drugs [FDA Orange Book]

FDA products linked to NCI funding through acknowledgment in patents listed in the New Drug Application (NDA).

Direct patent acknowledgements represent direct links to the NCI projects.

Direct acknowledgements by NDA listed patents identified 6 FDA approved drugs linked to 6 NCI funded projects.

SOURCES: Thomson Reuters Web of Science, MEDLINE, ScienceWire, The Patent Board, NIH IMPACII, FDA Orange Book, USPTO Issued Patents


11

Publication[MEDLINE]

Acknowledgement of NCI Support

Non-Patent Reference List

How has the NCI supported the development of FDA approved drugs? Methodology

Links established through non-patent references found in the patents listed in the NDA.

High confidence, direct links include those with the same personnel and organization in: • NDA listed patent• Publication it cites and • NCI Project that funded the

publication.

Automated approach identified high confidence, direct links: • 9 drugs linked to • 12 NCI funded projects.

NCI Project[IMPACII]

Patent Acknowledgement

of NCI Support

NDA-Listed Patent[USPTO Patent]

Approved Drugs [FDA Orange Book]



12

How has the NCI supported the development of FDA approved drugs? Findings

Drug Information Drug-Publication-Project Relationship Grant and Publication Data

Trade Name

Therapeutic Class

Patent Inventor, Publication Author & Project PI

Patent Assignee, Publication Affiliation & Project Organization

Project Start- Publication -Patent Filing

Grant Type Journal Citations

Cetrotide Gonadotropins, NEC Schally, Andrew Tulane University 1985-1988-1988 R01 PNAS 86

Clolar Montgomery, John Southern Research Institute 1983-1988-1994 P01 J. Med. Chem. 28

Folotyn Antineoplastic Agents, NEC Sirotnak, Francis Sloan-Kettering Institute for

Cancer Research 1993-1998-2005 R35 Cancer Chemoth. Pharm. 37

Thyrogen Kourides, Ione Sloan-Kettering Institute for Cancer Research 1977-1984-1993 R01 Recent Prog.

Horm. Res. 61

Trisenox Antineoplastic Agents, NEC Warrell, Raymond Sloan-Kettering Institute for

Cancer Research 1997-1998-2004 R03 Blood 187

Zinecard Speyer, James New York University 1975-1988-1992 P30‡ N. Eng. J. Med. 292

Zolinza Antineoplastic Agents, NEC Marks, Paul Sloan-Kettering Institute for

Cancer Research 1966 – 1989-1999 P30 PNAS 34

This table lists 7 of the 12 projects identified as having shared both person and organization information.Orange highlights indicate drug-to-project links established only through non-patent references (no direct patent reference to these projects)‡ P30CA16087 project was also identified as contributing to one of the breast cancer biomarkers



13

How has the NCI supported the development of FDA approved drugs? Portfolio Implications

• This analysis has shown that: • Non-Patent Reference data can serve as a source for additional

linkages to outputs like FDA approved drugs• Metadata about the outputs can be effectively used to identify high-

confidence, direct links

• Understanding NCI funding directly related to drug development helps to:• Measure late-stage impact of research funding on public health by

enabling analysis of drug-specific impacts• Potentially identify areas where support is needed for particular

indications or therapy targets


14

What can be done when the portfolio to be analyzed is not associated with output data?

• The International Cancer Research Partners includes 50 international funding agencies focused on cancer research who awarded over 36,000 research projects from 2006-2008.

• Although some members have developed systems for tracking outputs arising from ICRP research awards, the partnership does not yet have a comprehensive system.

• OSPA carried out a feasibility study of linking research publications in Thomson Reuters Web of Science to research projects in the ICRP portfolio.


15

Publication Outputs Missing Because Award Codes Were Insufficient to Match

• The feasibility study found that researchers were not acknowledging support by grant number (award code).

US DoD, CDMRP

American Cancer Society

Komen Breast Cancer Foun-

dation

California Breast Cancer

Research Program

Avon Founda-tion

Dutch Cancer Society

0

500

1,000

1,500

2,000

2,500U

niqu

e Aw

ard

Code

s on

ICRP

W

ebsit

e (2

006-

2011

)

5% 2% 10% 9% 1% 1%

Total ICRP AwardsAwards Matched to WoS Publications Percentage of Total ICRP Awards Matched to WoS Publications


16

Additional Matching Methods for Increasing Publication Output Identification

• Using Funding Agency Names Successful for funding agencies whose portfolio is exclusively focused on cancer

research (e.g. American Cancer Society), but not for broadly-focused portfolios (e.g. Wellcome Trust)

• Using Principal Investigator Name

SOURCES: Thomson Reuters Web of Science and ICRP

0

200

400

600

800

1,000

1,200

1,400

1,600ICRP PIsICRP Pis Matched to WoS Pubs

Un

iqu

e P

I Nam

es o

n IC

RP

Gra

nts

(200

6-20

11)


17

Identifying Candidate ICRP members using ICRP-funded Publications

Rank Funding Organization Pubs

1 National Natural Science Foundation of China 1488

2 Ministry of Education, Culture, Sports, Science and Technology of Japan 855

3 German Research Foundation 506

4 European Union 494

5 Italian Association for Cancer Research 489


1 Italian Association for Cancer Research 489

2 Swedish Cancer Society 429

3 German Cancer Aid 361

4 Danish Cancer Society 189

5 French Cancer Research Association 178


1 Japan Society for the Promotion of Science 354

2 Pfizer 239

3 Novartis 216

4 French Cancer Research Association 178

5 Leukemia & Lymphoma Society 177

Most Common

Cancer-specific Funders

Most Common

Funders

Most Common

Non-government Funders


18

Future development of outputs data for portfolio analysis

• Improve data collection and automation of weighted linkages.

• Establish data-driven linkages between funding and later stage research outputs and outcomes.

• Enable portfolio analysis across a broad range of data sources and methodologies to reflect the broad range of research outputs.


19

Acknowledgements

• Biomarkers- NCI OSPA: James Corrigan, Elizabeth Hsu, Lawrence Solomon - Discovery Logic/Thomson Reuters: Sandeep Patel, Tina Brust

• FDA Patent Analysis- NCI OSPA: James Corrigan, Elizabeth Hsu, Lawrence Solomon - Discovery Logic/Thomson Reuters: Duane Williams- The Patent Board: Kim Hamilton

• ICRP- NCI OSPA: James Corrigan, Elizabeth Hsu, Lawrence Solomon,

Samantha Finstad (AAAS Fellow)- Discovery Logic/Thomson Reuters: Sandeep Patel, Lauren Taffe

American Evaluation Association Annual Meeting, November 2011 Using Multiple Methods and Data...

Documents

Transcript of American Evaluation Association Annual Meeting, November 2011 Using Multiple Methods and Data...