Alex.Holcombe@sydney.edu.auSchool of Psychology

@ceptional

Fixing Science:The Replicability Crisis

http://www.slideshare.net/holcombea/

1Wednesday, 1 May 13

In the course of describing the scientific method, a popular second-grade textbook instructs students that "Experiments should be done more than once". But in the professional scientific literature, the proportion of published studies that report replications are very small. While individual research articles typically report statistics designed to insure that the false-positive rate is below a nominal value (e.g, .05), common forms of bias in scientific practice can inflate the false-positive rate to much higher levels. For this and other reasons, the second-graders are right: without replications, it is difficult to know whether to believe individual results.

Science now has the means to solve the problem, by taking advantage of new technology (the "Internet"). We are in a new era of cheap, open access, unlimited publishing. Nevertheless, academia continues to actively suppress replication studies. As this gradually changes, we will see not only a continued increase in the demand for statisticians, but also the emergence of a new class of data re-analysts. From my perspective as a psychological scientist, I will describe new initiatives that are coaxing replication studies, and raw data, out of the closet and onto the internet.

Abstract

2Wednesday, 1 May 13

The Replicability Crisis

Rule among early-stage venture capital firms that “at least 50% of published studies, even those in top-tier academic journals, can't be repeated with the same conclusions by an industrial lab” - Prinz, Schlange, & Asadullah. Nature Rev. Drug Discov. 10, 712 (2011)

Bayer HealthCare :only about 25% of published preclinical studies could be validated to the point at which projects could continue

Amgen Fi!y-three papers were deemed ‘landmark’ studies (see ‘Reproducibility of research "ndings’)... scienti"c "ndings were con"rmed in only 6 (11%) cases

3Wednesday, 1 May 13

Reasons for The Replicability Crisis

• Errors

• Fraud

• Publication bias by researchers

• Publication bias by journals

• Researchers p-hacking

Statistical Flukes

Statistical Flukes

}

4Wednesday, 1 May 13

http://www.nytimes.com/2013/04/19/opinion/krugman-the-excel-depression.html?_r=0

• Omitted some data

• Used unusual, questionable statistics

• Made an Excel error

ERRORS

5Wednesday, 1 May 13

Central to this initiative is a checklist intended to prompt authors to disclose technical and statistical information in their submissions and to encourage referees to consider aspects important for research reproducibility.

6Wednesday, 1 May 13

Why science is self-correcting

There's no point in scientific misconduct; it is always found.

Published on August 10, 2010 by Art Markman, Ph.D. in Ulterior Motives

Because scientists are always repeating each other's experiments, it is hard for a

fictitious result to hang on for very long.

“three unidentified young researchers as the whistleblowers for the case, and implies that these whistleblowers spent months making observations of Stapel and his work before they concluded that something actually was wrong” http://en.wikipedia.org/wiki/Diederik_Stapel

FRAUD

7Wednesday, 1 May 13

p-values

Compare two groups (of people/rats/molecules given different treatments).

Null hypothesis: That the treatment had no effect on the measure of interest (e.g., cancer rate)

Calculate the probability of the observed difference between the groups, assuming the treatment had no effect.

Is the probability less than .05? Then it’s statistically significant

Statistical Flukes

8Wednesday, 1 May 13

Publication Bias

bias introduced into the scientific literature by selective publication — chiefly by a tendency to publish positive results but not to publish negative or nonconfirmatory results.

9Wednesday, 1 May 13

http://xkcd.com/882/

10Wednesday, 1 May 13

Publication bias

• By journals

• “Unfortunately, despite the tens of thousands of available journals, places to send negative results are exceedingly scarce.”

• even for those that do, uphill battle to get accepted

• By researchers

• No glory (promotions, grants, prizes) for a negative result

http://expertedge.journalexperts.com/2013/04/26/negative-results-the-dark-

matter-of-research/

11Wednesday, 1 May 13

Corollary 4: The greater the flexibility in designs, definitions, outcomes, and analytical modes ina scientific field, the less likely the research findings are to be true. Flexibility increases the potential for transforming what would be “negative” results into “positive” results.

Corollary 6: The hotter a scientific field (with more scientific teams involved), the less likely the research findings are to be true.

Publication Bias

“In summary, while we agree with Ioannidis that most

research findings are false...”

12Wednesday, 1 May 13

Reasons for the Replicability Crisis

• Errors

• Fraud

• Publication bias by researchers

• Publication bias by journals

• Researchers p-hacking

Statistical Flukes

Statistical Flukes

}Partial solution: Conduct and publish replication studies

13Wednesday, 1 May 13

Publication bias: Replication studies•Difficult to publish non-replications and replications

•Most journals only publish papers that “make a novel contribution”

•Reviewers/editors tend to hold non-replicating manuscript to higher standard than original.

•Bem

•Little career incentive to publish a non-replication or a replication

http://www.flickr.com/photos/nickperez/2569423078 t. magnum

unpublished results

files

14Wednesday, 1 May 13

“Which of the following do you consider to be scientific fraud?

1) A scientist collects 100 observations in an experiment and discards the 80 that run counter to his desired outcome

2) A scientist runs ten experiments then selectively writes up the two that produced statistically significant effects

3) A journal reviews ten papers on the same topic and selectively publishes the two that reported statistically significant effects”

Dr. Chris Chambershttp://www.scilogs.com/sifting_the_evidence/tackling-the-f-word/

What’s fraud and what’s publication bias?

15Wednesday, 1 May 13

Reasons for the Replicability Crisis

• Errors

• Fraud

• Publication bias by researchers

• Publication bias by journals

• Researchers p-hacking

Statistical Flukes

Statistical Flukes

}Partial solution: Conduct and publish replication studies

16Wednesday, 1 May 13

Barriers to publishing replications and failed-replications

• No glory in publishing a replication

• Few journals publish replications

• usually uphill battle even with those that do

• The wrath of the original researcher

17Wednesday, 1 May 13

• problems: incentives

18Wednesday, 1 May 13

• problems: incentives

http://psychfiledrawer.org/view_article_list.php

Pashler, Spellman,

Holcombe& Kang (2011)

19Wednesday, 1 May 13

• problems: incentives

DETAILS page: http://psychfiledrawer.org/replication.php?attempt=MTU%3D

20Wednesday, 1 May 13

• problems: incentives

http://psychfiledrawer.org/view_article_list.php

Pashler, Spellman,

Holcombe& Kang (2011)

21Wednesday, 1 May 13

File-drawer fixes

• Journals that don’t reject replications as being uninteresting or unimportant

• Pre-registration of study designs and analysis methods

• Brief reporting of replications

✔•◦

✔

◦ ◦

◦◦

✔

22Wednesday, 1 May 13

• Collect data, check for statistical significance, collect more data

• Analyse multiple measures individually

23Wednesday, 1 May 13

• Pre-registration of study designs and analysis methods

24Wednesday, 1 May 13

File-drawer fixes

• Journals that don’t reject replications as being uninteresting or unimportant

• Pre-registration of study designs and analysis methods

• Brief reporting of replications

✔•◦

✔

◦ ◦

◦◦

✔

25Wednesday, 1 May 13

Registered Replication Reports

1. Authors plan a replication study

2. They submit an introduction and methods section

3. Sent to reviewers, including author of to-be-replicated article

4. Editor decides whether to accept/reject, based on:

1. Reviewer comments regarding the proposed protocol

2. Importance of original study, judged by argument in the introduction, number of citations of original, reviewer comments

5. The Intro, Method and analysis plan, and reviewer comments are posted on the journal website

6. After the results come in, the authors submit a conventional results and discussion section and that together with the raw data are posted, yielding the complete publication

1. some sort of minimal peer review needed for that. What exactly?

✔

Dan Simons

✔✔

http://www.psychologicalscience.org/index.php/replication

26Wednesday, 1 May 13

• Original author signed off on it, so can’t complain / hate the replication authors as much.

• Good way to start for a new PhD student, anyone planning to build on some already-published results

• Will post the raw data

• Will facilitate, publish meta-analyses when replications accrue

• Reduce the incentive to publish flashy, headline-grabbing but unreliable studies?

✔✔✔

Registered Replication Reports

27Wednesday, 1 May 13

Reasons for the Replicability Crisis

• Errors

• Fraud

• Publication bias by researchers

• Publication bias by journals

• Researchers p-hacking

Statistical Flukes

Statistical Flukes

}Partial solution: Conduct and publish replication studies

28Wednesday, 1 May 13

Open Research

• Post all experiment software online

• As data comes in, put on web

• Electronic lab notebook

• Post all data analyses and calculations

• Papers written via open collaborative documents on the web

The Tasman Declaration on Open Research

https://sites.google.com/site/nzauopenresearch/sign-up-to-the-tasman-declaration

29Wednesday, 1 May 13

Open Data

NHMRC: The next steps will be improving public and other researchers’ access to publicly funded data.

https://theconversation.edu.au/all-research-funded-by-nhmrc-to-be-accessible-free-of-charge-5486

30Wednesday, 1 May 13

• Errors

• Fraud

• Publication bias by researchers

• Publication bias by journals

• Researchers p-hacking

Statistical Flukes}Changing the landscape of science publishing:

Open access mandates, publisher boycotts, sharing science, and your CV

@ceptionalhttp://www.slideshare.net/holcombea/

The Replicability Crisis

31Wednesday, 1 May 13