US National Academies Study on Reproducibility in...

27
Xia & He Publishing US National Academies Study on Reproducibility in Scientific Results Harry Hua-Xiang Xia, Ph. D, M. Med Founder, Medjaden Bioscience (2005, Hong Kong; 2009, Mainland China; 2014, USA) Co-Founder, Xia & He Publishing (2011, Hong Kong; 2015, USA) Editor-in-Chief, Journal of Clinical and Translational Hepatology (Indexed by PubMed & ESCI) Adjunct Professor, First Affiliated Hospital, Guangdong Pharmaceutical University, Guangzhou, China President, Alliance for Scientific Editing in China (ASEC) May 31, 2018, Washington DC, USA 1

Transcript of US National Academies Study on Reproducibility in...

Page 1: US National Academies Study on Reproducibility in ...sites.nationalacademies.org/cs/groups/dbassesite/documents/webp… · Crisis or not: reproducibility and replicability in the

Xia & He Publishing

US National Academies Study on Reproducibility in Scientific Results

Harry Hua-Xiang Xia, Ph. D, M. MedFounder, Medjaden Bioscience (2005, Hong Kong; 2009,

Mainland China; 2014, USA)Co-Founder, Xia & He Publishing (2011, Hong Kong; 2015, USA)Editor-in-Chief, Journal of Clinical and Translational Hepatology

(Indexed by PubMed & ESCI)Adjunct Professor, First Affiliated Hospital, Guangdong

Pharmaceutical University, Guangzhou, ChinaPresident, Alliance for Scientific Editing in China (ASEC)

May 31, 2018, Washington DC, USA 1

Page 2: US National Academies Study on Reproducibility in ...sites.nationalacademies.org/cs/groups/dbassesite/documents/webp… · Crisis or not: reproducibility and replicability in the

Xia & He Publishing

1985-1990, Union Hospital, Tongji Medical College, HUST, Wuhan

1990-1995, St. James Hospital, Trinity College, Dublin University, Dublin

1995-2000, Nepean Hospital, The University of Sydney, Sydney

2000-2006, Queen Mary Hospital, The University of Hong Kong, Hong Kong

2006-2014, Novartis Pharmaceuticals Corporation, East Hanover, NJ

2014-Present, Medjaden Bioscience (Hong Kong,China)Xia & He Publishing, TX 2

My Research and Academic Career

Page 3: US National Academies Study on Reproducibility in ...sites.nationalacademies.org/cs/groups/dbassesite/documents/webp… · Crisis or not: reproducibility and replicability in the

Xia & He Publishing

Q1a: Are there specific examples in your country/region where a lack of reproducibility and replicability in research results has led to doubt about reported results more broadly?

www.nature.com/news/replications-ridicule-and-a-recluse-the-controversy-over-ngago-gene-editing-intensifies-1.20387

Page 4: US National Academies Study on Reproducibility in ...sites.nationalacademies.org/cs/groups/dbassesite/documents/webp… · Crisis or not: reproducibility and replicability in the

Xia & He Publishing

Q1a: Are there specific examples in your country/region where a lack of reproducibility and replicability in research results has led to doubt about reported results more broadly?

www.nature.com/news/replications-ridicule-and-a-recluse-the-controversy-over-ngago-gene-editing-intensifies-1.20387

Page 5: US National Academies Study on Reproducibility in ...sites.nationalacademies.org/cs/groups/dbassesite/documents/webp… · Crisis or not: reproducibility and replicability in the

Xia & He Publishing

Crisis or not: reproducibility and replicability in the scientific world

More than 70% of researchers have tried and failed to reproduce another scientist's experiments;

More than 50% have failed to reproduce their own experiments;

52% of those surveyed agree that there is a significant 'crisis' of reproducibility;

Less than 31% think that failure to reproduce published results means that the results are probably wrong.

Q1b: Are reproducibility and replication of research results a global concern or is it a concern focused within specific countries?

Baker M. 1,500 scientists lift the lid on reproducibility: Survey sheds light on the ‘crisis’ rocking research. Nature, 2016 May, 533(7604):452-4

Page 6: US National Academies Study on Reproducibility in ...sites.nationalacademies.org/cs/groups/dbassesite/documents/webp… · Crisis or not: reproducibility and replicability in the

Xia & He Publishing

Reasons for the failure of reproducibility and replicability

“Publish or perish” research environment;

Insufficient training for the researchers (studyprocedures, statistics, etc.);

Selective use of data, inappropriate study conditions, etc.

Academic misconduct: Fraud (Fabrication, falsification, etc.)

Q1b: Are reproducibility and replication of research results a global concern or is it a concern focused within specific countries?

Page 7: US National Academies Study on Reproducibility in ...sites.nationalacademies.org/cs/groups/dbassesite/documents/webp… · Crisis or not: reproducibility and replicability in the

Xia & He Publishing

Analyses of retractions of scientific publications

Q1b: Are reproducibility and replication of research results a global concern or is it a concern focused within specific countries?

Figure 1(A) Number of retracted

articles for specific causes by year of retraction.

(B) Percentage of published articles retracted for fraud or suspected fraud by year of publication.

Fang F. Misconduct accounts for the majority of retracted scientific publications PNAS 2012;109(42):17028

Page 8: US National Academies Study on Reproducibility in ...sites.nationalacademies.org/cs/groups/dbassesite/documents/webp… · Crisis or not: reproducibility and replicability in the

Xia & He Publishing

Q1b: Are reproducibility and replication of research results a global concern or is it a concern focused within specific countries?Analyses of retractions of scientific publications

Figure 2. Country of origin of publications retracted for fraud or suspected fraud (A), plagiarism (B), or duplicate publication (C).

Fang F. Misconduct accounts for the majority of retracted scientific publications, PNAS. 2012;109(42):17028

Page 9: US National Academies Study on Reproducibility in ...sites.nationalacademies.org/cs/groups/dbassesite/documents/webp… · Crisis or not: reproducibility and replicability in the

Xia & He Publishing

Q1b: Are reproducibility and replication of research results a global concern or is it a concern focused within specific countries?

Rank CountryPapers retracted

Retraction rate(per 10,000)

1 USA 1065 2.92 China 313 1.83 UK 174 2.04 Germany 167 1.75 Japan 162 2.06 India 162 3.47 The Netherland 117 3.48 Canada 109 1.89 Italy 102 1.810 Australia 88 1.9

August 2010-November 11, 2016 http://retractionwatch.com/

Retraction by countries

Page 10: US National Academies Study on Reproducibility in ...sites.nationalacademies.org/cs/groups/dbassesite/documents/webp… · Crisis or not: reproducibility and replicability in the

Xia & He Publishing

Q1b: Are reproducibility and replication of research results a global concern or is it a concern focused within specific countries?

Retraction by individual authors

November 11, 2016 http://retractionwatch.com/the-retraction-watch-leaderboard/

Yoshitaka Fujii (藤井 善隆)A Japanese report concludes: “It is as if someone sat at a desk and wrote a novel about a research idea.”

Page 11: US National Academies Study on Reproducibility in ...sites.nationalacademies.org/cs/groups/dbassesite/documents/webp… · Crisis or not: reproducibility and replicability in the

Xia & He Publishing

Q1b: Are reproducibility and replication of research results a global concern or is it a concern focused within specific countries?Dutch former professor of social psychology

Page 12: US National Academies Study on Reproducibility in ...sites.nationalacademies.org/cs/groups/dbassesite/documents/webp… · Crisis or not: reproducibility and replicability in the

Xia & He Publishing

Q1b: Are reproducibility and replication of research results a global concern or is it a concern focused within specific countries?

Hwang Woo-suk (黄禹锡), South KoreaSCNT - Somatic cell nuclear transferHwang WS, et al. Evidence of a pluripotent human embryonic stem cell line derived from a cloned blastocyst. Science. 2004 Mar 12; 303 (5664): 1669–74. doi:10.1126/science.1094515. PMID 14963337. (Retracted, see PMID 16410485)

Haruko Obokata (小保方 晴子), JapanSTAP - Stimulus-triggered acquisition of pluripotency1. Obokata H, et al. Stimulus-triggered fate conversion of somatic cells

into pluripotency. Nature. 2014 Jan 30;505(7485):641-7. doi: 10.1038/nature12968. Retraction in: Nature. 2014 Jul 3;511(7507):112. PMID: 24476887

2. Obokata H, et al. Bidirectional developmental potential in reprogrammed cells with acquired pluripotency. Nature. 2014 Jan 30;505(7485):676-80. doi: 10.1038/nature12969. Retraction in: Nature. 2014 Jul 3;511(7507):112. PMID: 24476891

Page 13: US National Academies Study on Reproducibility in ...sites.nationalacademies.org/cs/groups/dbassesite/documents/webp… · Crisis or not: reproducibility and replicability in the

Xia & He Publishing

Q1b: Are reproducibility and replication of research results a global concern or is it a concern focused within specific countries?

Based on the above evidence, I believe that reproducibility and replication of research results are a global concern!

Page 14: US National Academies Study on Reproducibility in ...sites.nationalacademies.org/cs/groups/dbassesite/documents/webp… · Crisis or not: reproducibility and replicability in the

Xia & He Publishing

Our research on perceptions of Chinese medicalresearchers on misconduct in China

Liao QJ, Zhang YY, Fan YC, Zheng MH, Bai Y, Eslick GD, He XX, Zhang SB, Xia HH, He H. Perceptions of Chinese Biomedical Researchers Towards Academic Misconduct: A Comparison Between 2015 and 2010. Sci Eng Ethics. 2018 Apr;24(2):629-645. doi: 10.1007/s11948-017-9913-3

Page 15: US National Academies Study on Reproducibility in ...sites.nationalacademies.org/cs/groups/dbassesite/documents/webp… · Crisis or not: reproducibility and replicability in the

Xia & He Publishing

5.3

37.6

28.3

19.7

9.0

3.9

42.3

32.6

14.7

6.6 4.4

40.7

31.1

16.4

7.4

0.0

10.0

20.0

30.0

40.0

50.0

Not serious Somewhat serious Serious Very serious Extremely serious

Perc

enta

ge (%

)

Current status of misconduct

2010

2015

total

Our research: perceptions on current academic misconduct in Mainland China

Overall, 55% of participants thought that academic misconduct was serious, very serious or extremely serious.

Page 16: US National Academies Study on Reproducibility in ...sites.nationalacademies.org/cs/groups/dbassesite/documents/webp… · Crisis or not: reproducibility and replicability in the

Xia & He Publishing

22.87

12.78

26.56

15.87

25.27

14.7916.82

33.63

20.07

30.05

18.94

31.30

17.49

24.44

19.23

27.64

18.62

26.53

9.64

16.82

7.45

13.70

8.22

14.79

30.94

10.99

25.96

12.26

27.70

11.82

0.00

10.00

20.00

30.00

40.00

1st Choice 2nd Choice 1st Choice 2nd Choice 1st Choice 2nd Choice

2010 2015 total

The most common form of misconduct

Plagiarism Falsification Fabrication Duplication Authorship

Our research: most common forms of academic misconduct in mainland China

Impropriate authorships and plagiarism were the most common forms of academic misconduct, followed by falsification and fabrication.

Page 17: US National Academies Study on Reproducibility in ...sites.nationalacademies.org/cs/groups/dbassesite/documents/webp… · Crisis or not: reproducibility and replicability in the

Xia & He Publishing

Our research: reasons behind academic misconduct in the Mainland China

52.69

31.61

49.04

31.01

50.31

31.22

7.40

47.53

9.13

43.39

8.53

44.84

33.18

13.23

34.13

16.83

33.80

15.57

0.45 2.020.24 0.96 0.31 1.33

4.26 3.146.13 7.33

5.48 5.87

0.00

20.00

40.00

60.00

1st Choice 2nd Choice 1st Choice 2nd Choice 1st Choice 2nd Choice

2010 2015 total

Reasons behind misconduct

National system Institutional system Social environment Family influence Individual behaviors

The most important reason behind academic misconduct was the academic/research assessment system, followed by the social environment

Page 18: US National Academies Study on Reproducibility in ...sites.nationalacademies.org/cs/groups/dbassesite/documents/webp… · Crisis or not: reproducibility and replicability in the

Xia & He Publishing

Our research: estimation of percentage of publication involving academic misconduct in Mainland China

Participants estimated that 40.1% (39.8 ± 23.5% in 2010; 40.2 ± 24.5% in 2015) of published scientific articles were associated with some form of academic misconduct.

Year Answer (%)

0-20% 20.1-40% 40.1-60% 60.1-80% 80.1-100%

2010 29.9 27.2 25.8 13.2 3.9

2015 28.9 27.9 25.5 13.1 4.7

Total 29.2 27.6 25.6 13.1 4.5

Percentage of publication involving misconduct

Page 19: US National Academies Study on Reproducibility in ...sites.nationalacademies.org/cs/groups/dbassesite/documents/webp… · Crisis or not: reproducibility and replicability in the

Xia & He Publishing

Our research: Conclusions

Conclusions The perceptions of Chinese biomedical

researchers towards academic misconduct had not significantly changed from 2010 to 2015.

Reform of the academic assessment system should be the fundamental approach to tackling this problem in China.

Page 20: US National Academies Study on Reproducibility in ...sites.nationalacademies.org/cs/groups/dbassesite/documents/webp… · Crisis or not: reproducibility and replicability in the

Xia & He Publishing

Q2 Are there particular scientific fields in which lack of reproducibility and replicability is more/less of a concern?

More Life Science;Medicine

Less Engineering Business

Page 21: US National Academies Study on Reproducibility in ...sites.nationalacademies.org/cs/groups/dbassesite/documents/webp… · Crisis or not: reproducibility and replicability in the

Xia & He Publishing

Measures taken by Chinese government to tackle academic misconduct (but not specifically reproducibility and replicability)

Suspend research projects;

Cancel grants and rescind promotion;

Demote, or dismiss from the office;

Reform in the academic assessment system

Q3 Are there any concrete actions that organizations (e.g. funders, publishers, societies) in your country or region have taken to address concerns about reproducibility and replicability? What actions should they take?

http://www.moe.edu.cn/srcsite/A02/s5911/moe_621/201607/t20160718_272156.html

Page 22: US National Academies Study on Reproducibility in ...sites.nationalacademies.org/cs/groups/dbassesite/documents/webp… · Crisis or not: reproducibility and replicability in the

Xia & He Publishing

Proposed actions to tackle the problem Generate the ”reproducibility index” for journals, similar

to an impact factor;

Reproducibility index = ?

Regulate funding bodies

Improve training for researchers

Radom results audits

Q3 Are there any concrete actions that organizations (e.g. funders, publishers, societies) in your country or region have taken to address concerns about reproducibility and replicability? What actions should they take?

https://theconversation.com/science-is-in-a-reproducibility-crisis-how-do-we-resolve-it-16998

Page 23: US National Academies Study on Reproducibility in ...sites.nationalacademies.org/cs/groups/dbassesite/documents/webp… · Crisis or not: reproducibility and replicability in the

Xia & He Publishing

Yes, both regionally and globally.

Priorities

1) Modify the academic assessment system to be less publication-orientated;

2) Increase the awareness of prevention of academic misconduct through courses, workshop and training;

3) Increase the transparency of doing research, which can be referred as open science.

Q4 Should the research community work regionally and/or globally to address concerns about reproducibility and replicability? If so, what should be the priorities?

https://theconversation.com/science-is-in-a-reproducibility-crisis-how-do-we-resolve-it-16998

Page 24: US National Academies Study on Reproducibility in ...sites.nationalacademies.org/cs/groups/dbassesite/documents/webp… · Crisis or not: reproducibility and replicability in the

Xia & He Publishing

A Comprehensive Guide to English Medical Manuscript Writing and Publication

Huaxiang Xia, Yuanyuan Zhang. A Comprehensive Guide to English Medical Manuscript Writing and Publication. Wuhan: HUST Press, 2017

Page 25: US National Academies Study on Reproducibility in ...sites.nationalacademies.org/cs/groups/dbassesite/documents/webp… · Crisis or not: reproducibility and replicability in the

Xia & He Publishing

A Comprehensive Guide to English Medical Manuscript Writing and Publication

Huaxiang Xia, Yuanyuan Zhang, A Comprehensive Guide to English Medical Manuscript Writing and

Publication HUST Press, Wuhan, 2017

Page 26: US National Academies Study on Reproducibility in ...sites.nationalacademies.org/cs/groups/dbassesite/documents/webp… · Crisis or not: reproducibility and replicability in the

Xia & He Publishing

Acknowledgments

Selin Hua He. M. PhilCEO MedjadenXia & HE Publishing

Sarah Yuanyuan Zhang, Ph. DAcademic DirectorMedjadenXia & HE Publishing

Page 27: US National Academies Study on Reproducibility in ...sites.nationalacademies.org/cs/groups/dbassesite/documents/webp… · Crisis or not: reproducibility and replicability in the

Xia & He Publishing

Welcome to China

The Second International Summit on Medical Research and Publication

September 16, 2018, Wuhan, ChinaHosted by Xia & He Publishing, Medjaden & Tongji Medical College

http://www.xiahepublishing.com/MeetingDetail.aspx?sid=3&id=5