Northern Greece Diabetes Meeting -...

Irreproducibility of published research in the

biosciences: From etiology to therapeutics.

Northern Greece Diabetes MeetingNov 19, 2016

Jeffrey S. Flier, M.D.

Irreproducibility: The Big Picture(basic and preclinical research)

Recent progress in biomedical sciences has been remarkable –- understanding and treatment of human disease.


Recent progress in biomedical sciences has been remarkable –- understanding and treatment of human disease.

Growth in magnitude, scope, professionalization

› 1,000,000 publications/yearThousands of journals exist

› $100B/year in spent on biomedical research in the U.S.

Very complex ecosystem


There will always be errors, dead ends, mistaken claims and other problems;

- the “self-correcting” nature of science is expected to deal with these.

Perfect reproducibility is both impossible and undesirable.

If we demanded perfect reproducibility, progress wouldbe reduced.


However………

There is Increasing evidence that bioscience research published today

has a level of irreproducibility that is excessive and alarmingly high.


However………

There is increasing evidence that bioscience research published today has a

level of irreproducibility that is excessive and alarmingly high.

-- Some (John Ioannidis) claim more than 75% of papers are false.


However………

There is increasing evidence that bioscience research published today has a

level of irreproducibility that is excessive and alarmingly high.

-- Some claim more than 75% of papers are false.

If true, this fact would diminish progress from investment in research, and

threaten the confidence in research by funders and the public.

My Perspective on These Issues

• 40 years as physician scientist

• Role in publishing as reviewer and editor

• Academic leadership

• Dean of Harvard Medical School (9 years, recently completed term)

9,500 full time faculty – promotions and appointments

Large and robust research program

Oversaw investigations of research misconduct

Occasional spokesperson for biomedical research

Data Reproducibility: Case Study #1 : Betatrophin

Hypothesis: Severe Insulin Resistance causes beta cell hyperplasia -evidence that a circulating factor is involved. (S. Flier, Kulkarni, Kahn - 2001).


Hypothesis: Severe Insulin Resistance causes beta cell hyperplasia -evidence that a circulating factor is involved. (S. Flier, Kulkarni, Kahn - 2001).

D. Melton lab (chair of Harvard stem cell dept) reports a new hormone - “betatrophin”

- mRNA massively induced in liver of mice with severe insulin resistance;- administering the encoded protein –”betatrophin”- induces marked beta cell

hypertrophy in treated mice (Cell, 2013).

But then, one year later ……………………………

Gusarova et al , Cell 2014 – “betatrophin does not control beta cell expansion” ;

- repeated experiment with no beta cell response; knock out mouse lacking “betatrophin” has normal hypertrophy in response to insulin resistance.


Melton et al, Cell 2014 – “Editorial” on betatrophin –states his lab could not confirm the activity they previously reported.

Melton et al PLoS One 2016 – multi lab study further confirming that betatrophic activity in 2013 Cell paper could not be reproduced.

The Bad – Prominent scientist; high profile journal; major claim……….But that major claim is false.

The Good – quick resolution of the error; no disagreement remains; no long term damage was done to the field.

The Questions – What accounts for the mistake having been made? In Melton lab; At the journal that published the paper - Cell.

Data Reproducibility: Case Study #2 : STAP Cells

Stem Cell biology has generated great excitement and therapeutic potential

Approaches to making pluripotent stem cells – Nobel prize to Yamanaka in 2012for technique to make pluripotent stem cells from skin cells using transcription factors.




Jan 2014 – Paper in Nature by Obotaka et al – STAP cells described – “Stimulus-triggered acquisition of pluripotency”

– exposure of skin cells to weak acid makes them pluripotent!(Collaboration between scientists at RIKEN in Japan and Brigham and Womens/HMS)




Jan 2014 – Paper in Nature by Obotaka et al – STAP cells described – “Stimulus-triggered acquisition of pluripotency” – exposure of skin cells to weak acid makes them pluripotent!(Collabortion between RIKEN in Japan and Brigham and Womens/HMS)

Later in 2014 – Many labs unable to reproduce this;- RIKEN conducts investigation, finds Obotaka committed fraud; - Nature retracts paper in July 2014 (6 months after publication)




Jan 2014 – Paper in Nature by Obotaka et al – STAP cells described – “Stimulus-triggered acquisition of pluripotency” – exposure of skin cells to weak acid makes them pluripotent!(Collabortion between RIKEN in Japan and Brigham and Womens/HMS)

Later in 2014 – Many labs unable to reproduce this; - RIKEN conducts investigation, finds Obotaka committed fraud; - Nature retracts paper in July 2014

Sasai, Obotaka advisor and co-author at RIKEN commits suicide in Aug, 2014.

Data Reproducibility: Case Study # 3, 4, n……..

Papers in Cell, Science, Nature etc……. +/- attention by the press

Broad interest in the work, but no clear confirmations or refutationsare published.

Scientists in academia and industry informally discuss their inability to reproducethe work.

Discussion and publication on the topic diminishes, or continues amidst confusion.

How should we view these findings pending further study?: irreproducible, contested, uncertain, or just irrelevant?

Reproducibility of Clinical Research

I have focused on basic science and preclinical research – but the problemof reproducibility is of equal or greater magnitude in clinical research.

Many studies are:

- poorly designed and underpowered - employ faulty statistic approaches and reagents- suffer from investigator biases that are not controlled forpublication bias favors “positive results” – difficulty publishing negative studies

Because these publications involve human subjects, the implications of irreproducibility are even more alarming………..

Research and Publishing Ecosystem

Scientist

Idea

Resources/Institution


Scientist

Idea


Research Results

Internal discussionsBroader presentationsPossible grant proposal


Scientist

Idea


Research ResultsSubmission forPublication

Peer Review

Editorial decision



Scientist

Idea


Research ResultsSubmission forPublication

Peer Review

Editorial decision


Reject

New submission

Accept!

Reconsider withAdditional work

RejectAccept!

Accept!

It’s Not Finished With Acceptance

Accept!

Part ofScientific Literature

Engagement viaScientific communityOver time


Accept!



Impact/meaning ofPublished work confirmed


Accept!



Press



Accept!



Press

Academic CareersPromotionsGrantsFame and fortune

Impact/meaning ofPublished work confrmed


Accept!



Press



Impact/meaning ofPublished work denied


Accept!


Post Publication Review PubPeer, others


Press




Irreproducibility assertedAnd/or proven


Accept!




Press





Investigation

Possible correctionOr retraction

? Sustained Uncertainty


Not a simple issue – the definition is ambiguous.

Precise replication is the most stringent, but least common approach. This

may be required if claim is important, doubt about finding, high profile,

many labs working in the area (betatrophin, STAP cells).

If finding is robust, it may be seen as “true” even without precise replication;

it is a result that can be productively built on by others.

If finding not robust, precise replication might be required to determine its’

truth, but its’ importance may be seen as diminished.

What does reproducibility of research mean?

- Fully stands the test of time –multiply reproduced and

extended; part of “accepted canon” – (though can always be

reinterpreted or modified by future work)

Irreproducibility of Published Research: The Spectrum.

- Fully stands the test of time –multiply reproduced and

extended; part of “accepted canon” – (though can always be

reinterpreted or modified by future work)

- “Some elements” true, but others, including possibly

important ones, shown to be erroneous based on additional

work. These issues may or may not “change the major

conclusions”, but remain a concern.


- Fully stands the test of time – reproduced and extended; part of

“accepted canon” – (can always be reinterpreted or modified)

- “Some elements” true, but others, including possibly important

ones, seen as erroneous based on additional work. These issues

may or may not “change the major conclusions”, but remain a

concern.

- Viewed as largely or completely untrue – diverse causes:

- honest errors/incompetence

- misconduct, fraud (less common).


- Fully stands the test of time – reproduced and extended; part of

“accepted canon” – (can always be reinterpreted or modified)

- “Some elements” true, but others, including possibly important

ones, seen as erroneous based on additional work. These issues

may or may not “change the major conclusions”, but remain a

concern.

- Viewed as largely or completely untrue – diverse causes:

- honest errors/incompetence

- misconduct, fraud (less common).


Much published work is rarely read, cited, or subject to reproduction, so whether it is ultimately reproducible is unknown.

The Spectrum of Irreproducibility of Published Research

Completelyreproducible

Largelyreproducible

Largelyirreproducible

Completelyirreproducible

How Common?Why?

- We really don’t know – many high estimates - different definitions;

how to assess?

Irreproducibility: How common is it?

- We really don’t know – many high estimates - different definitions;

how to assess?

- Appears to be increasing – as seen by research “watchdogs”, the press,

pharma, funding agencies –


Some Data on Reproducibility of Preclinical Research in High Impact Journals

Cost and Categories of Irreproducible Preclinical Research

New “Science of Reproducibility”, and New Watchdogs

Ivan Oransky

- more common in high profile journals

( ? more attention to those papers)

Are Some Journals More Irreproducible?

- number of retractions is clearly increasing over past 15 years –

does this represent more scrutiny, better tools to uncover

manipulation vs change in underlying rate?

Irreproducibility: Retractions Rising

? Tip of the iceberg

- Poor training/ experimental methodology – design, statistics, inference;

Observer bias, lack of double blind or internal replication

- Poorly characterized reagents – antibodies (? Half of analytic antibodies

give false/non specific results), inhibitors, cell lines

- Complex, interdisciplinary collaborations diminish understanding of

project elements by principal investigator (“I didn’t understand that

part”)

- Poor oversight/mentorship re: integrity of data and its presentation

(selective reporting)

- Deficiencies within the peer review/publishing ecosystem

Irreproducibility: Causes (1)

- Inappropriate response to external incentives –

grants, promotions, fame, financial gain

“cutting corners”, selective data reporting, etc

blend into or lead to more explicit manipulation and misconduct.

Irreproducibility: Causes (2)

- Ethical lapses sociopathy -

The most extreme cases can be hard to identify --- results “too good to be true”Be skeptical, don’t hesitate to have others confirm -

How Do Journals Contribute to the Problem?

Criteria for publication – impact more than the truth.

Excessive interest in Novelty/impact/Buzz

Lack of interest by publisher in confirmation, refutation, or important extensions of published work -- “publication bias”

Editorial requests for more and specific data – dangerous incentives to provide “required experimental results”, tempting susceptible scientists.

Changes in Scientific Publishing That Might Enhance Reproducibility of Published Work

(and improve the quality of peer review/editorial decision-making)

Changes to Peer Review –

- Make reviews of papers publically available and citable

- Identify reviewers to authors and community

- All versions of accepted papers kept online

- Availability of raw data for independent analysis

Goal : Accountability; higher quality; less bias

What can academia do about the problem?

- Accept that it is a problem, and get serious about addressing it.

- Some successful scientists and institutions have too little interest

in this issue – they think the system serves them well.

- Enhance access to and quality of training in experimental

design/stats/research and publication ethics.

What can academia do about the problem?

- Attempt to modify the culture of research. (This is hard to do).

More rewards/respect for those who produce reproducible

results, and those who take responsibility when their work does not

meet these standards; fewer rewards/respect for those who do not.

- culture responds more to actions, than to words.

- Reflect this in approach to academic appointments, promotions,

awards, etc -

( develop “Reproducibility index” to complement the H index……… )

Rx for Irreproducibility: The Ecosystem

Scientist andResearch culture

Institutional Culture andExpectations

Funders of Research

Biotech andPharma

PublishingEcosystem

The Pressand

Social Media

Two main goals: 1) Aligning incentives 2) Accountability

May Your Research be Both Important and Reproducible!

Thank You for inviting me to the lovely city of Thessaloniki!!

Research and Publishing EcosystemScientist

Idea


Research Results Submission forPublication

? Archive

Peer Review

Editorial decision


Reject

New submissionAccept!

Reconsider withAdditional work

RejectAccept!

Part ofScientific Literature Post Publication Review

PubPeer, others


Press


Impact/meaning ofPublished work secured

Impact/meaning of

Published work deniedIrreproducibility assertedAnd/or proven

Investigation


Sustained Uncertainty

+2000 journals400,000 pub articles1.3 M submitted

Accept!




Press


Impact/meaning ofPublished work secured



Investigation


? Sustained Uncertainty

It’s Not Over With Acceptance

- We really don’t know – many estimates - different definitions;

how to assess?

- Perception that it is increasing – expressed by watchdogs, press,

pharma, funders – Amgen paper on preclinical cancer research

as example

- ? more common in high profile journals vs more attention to

those papers (many papers rarely read, referenced).

- number of retractions is clearly increasing over past 15 years –

does this represent more scrutiny, better tools to uncover

manipulation vs change in underlying rate?


The Publishing Ecosystem: The Players

Publishers : Scientific/Professional Societies; For profit publishers; many new models (e-life)

Editors – scientific leaders vs professional editors

Reviewers – domain experts; unpaid

Peer review models –Closed (anon reviewers, hidden reviews, final version published) vs Open ( reviewers identified, reviews published, multiple versions)

Financial models – subscription vs open access with fees to publish

Print vs digital

- Can be slower than ideal to publish results for scientific community to

evaluate.

- Publication bias – disinterest in negative data, confirmation, “incomplete

stories” ; exaggerated interest in claims of novelty

- May favor the work of some scientists over others independent of the

quality/importance of the work – “old boys network”

- Expensive; inadequate public access

What are some flaws of the current publishing ecosystem?

- May favor the work of some scientists over others independent of the

quality/importance of the work – “old boys network”

- Does not optimize quality and value of the review process to the authors or

the reviewers.

- At a time of digital access and interconnectedness, it should be possible to

disrupt the current system to produce outcomes more valuable to the

scientists, their institutions, and funders – and eventually the public at large.

What are some flaws of the current publishing ecosystem?

Northern Greece Diabetes Meeting -...

Documents

Transcript of Northern Greece Diabetes Meeting -...