Fraud and Why Studies are Flawed: Should Journalists Trust Peer Review?

Click here to load reader

Embed Size (px)

Transcript of Fraud and Why Studies are Flawed: Should Journalists Trust Peer Review?

  1. 1. Fraud and Why Studies are Flawed:Should Journalists Trust Peer Review?Ivan Oransky, MDCo-Founder, Retraction Watch Executive Editor, Reuters Health Scripps Howard Institute onthe Environment and ScienceMay 23, 2012
  2. 2. Can You Trust Journal Studies? How good is peer review? Positive publication bias Over-reliance on embargoed studies How often it turns out to be wrong How to get it right
  3. 3. How Does Peer Review Work? Usually three outside reviewers Usually anonymous Sometimes in as little as 72 hours
  4. 4. How Good Is Peer Review?
  5. 5. How Good Is Peer Review?
  6. 6. How Good Is Peer Review? Bad at detecting fraud Slow Costly (and unfunded) Prone to bias
  7. 7. Peer Reviewers: Worse With Experience
  8. 8. Positive Publication Bias
  9. 9. Positive Publication BiasPublish a trial that will bring US$100,000 ofprofit or meet the end-of-year budget byfiring an editor. -- Former BMJ editor Richard Smith
  10. 10. Embargoes and the Ingelfinger RuleBy the late 20th century, journals needed to competenot just with each other but with newspapers and othermediaIn 1969, the Journal articulated this relationshipin its Ingelfinger Rule, a policy against publishinganything that had already appeared elsewhere. Otherjournals followed suit. This rule, combined withembargo policies, has led to a carefully choreographedproduction in which medical journals and the popularpress work cooperatively and competitively toinfluence the news cycle.-- NEJM, April 19, 2012
  11. 11. Even Without Embargoes, Wed Still HaveIngelfinger
  12. 12. How Often Are Studies Wrong? Ioannidis JPA. PLoS Med 2005; 2(8): e124
  13. 13. How Often Are Studies Wrong?
  14. 14. Retraction Watch Launched August 2010 with Adam Marcus Frequently cited in major news outlets, from Natureto Der Spiegel to New York Times 250,000 pageviews per month
  15. 15. Retractions on the Rise -The Wall Street Journal
  16. 16. Retractions on the Rise
  17. 17. Retractions on the Rise -Neil Saunders
  18. 18. The Unofficial Record Holder
  19. 19. Why Do Journals Retract? -Nature
  20. 20. Why Do Journals Retract? Error is more common than fraud 73.5% of papers retracted for error (orundisclosed reason) vs 26.6% for fraud Most common cause of retraction: scientificmistake (234 papers; 31.5%) Fabrication (including data plagiarism) morecommon than text plagiarism 67 retractions (9.0%) had multiple causes, but 134papers (18.1%) retracted for ambiguous reasons -Journal of Medical Ethics 2010
  21. 21. Fraud: Image Manipulation
  22. 22. Fraud: Faked Data
  23. 23. Is Fraud on the Rise?Over the yearssurveys have asked scientistsdirectly about their behaviour. [T]hese studieshave used different methods and asked differentquestions, so their results have been deemedinconclusive and/or difficult to compare. A non-systematic review based on survey and non-surveydata led to estimate that the frequency of seriousmisconduct, including plagiarism, is near 1%. -- Fanelli, PLoS ONE, 2009
  24. 24. Is Fraud on the Rise?A pooled weighted average of 1.97% (N = 7, 95%CI:0.864.45) of scientists admitted to havefabricated, falsified or modified data or results atleast once a serious form of misconduct by anystandard and up to 33.7% admitted otherquestionable research practices. In surveys askingabout the behaviour of colleagues, admission rateswere 14.12% (N = 12, 95% CI: 9.9119.72) forfalsification, and up to 72% for other questionableresearch practices.-- Fanelli, PLoS ONE, 2009
  25. 25. Is Fraud on the Rise?Meta-regression showed that self reportssurveys, surveys using the words falsificationor fabrication, and mailed surveys yieldedlower percentages of misconduct. When thesefactors were controlled for, misconduct wasreported more frequently bymedical/pharmacological researchers thanothers. -- Fanelli, PLoS ONE, 2009
  26. 26. Is Fraud on the Rise?Considering that these surveys ask sensitivequestions and have other limitations, it appearslikely that this is a conservative estimate of thetrue prevalence of scientific misconduct.-- Fanelli, PLoS ONE, 2009
  27. 27. This is Transparency?
  28. 28. Should Reporters Cover Conferences?
  29. 29. Conference Pitfalls Conferences select presenters based on < 1000 words Urologists at U of Florida & Indiana U studied 126randomized controlled trials presented in 2002-2003
  30. 30. Conference Pitfalls RCTs are the gold standard of medical evidence But the quality of that evidence wasnt pretty No abstract said how trial subjects were randomlyassigned to different treatments or placebos None told how the study ensured that neither theresearchers nor their doctors knew which they got Only about a quarter said how long researchersfollowed the subjects in the trial
  31. 31. Just Say NoSometimes, its better not to cover something.But if you must
  32. 32. Always Read the StudyWriting about a study after reading just apress release or an abstract without reading the entire paper is journalistic malpractice
  33. 33. How to Get Studies for embargoed material Association of Health Care Journalists membershipincludes access to Cochrane Library, Health Affairs,JAMA, and many other ScienceDirect (Elsevier) gives reporters free access tohundreds of journals Open access journals (e.g., Public Library of Ask press officers, or the authors
  34. 34. Who Has an Interest? Disclose conflicts Dollars For Docs series
  35. 35. Dont Rely Only on Study Authors Find outside sources. Heres how:
  36. 36. A Dirty Little SecretKeep a biostatistician in your back pocket Photo by Peyri Herrera, on Flickr
  37. 37. Acknowledgement/ContactThanks: Nancy Lapid, Reuters Health Twitter: @ivanoransky