How Many Scientists Fabricate and

Falsify Research?

A Systematic Review and Meta-Analysis

of Survey Data

Daniele Fanelli

““The million dollar question”….The million dollar question”….

Data source Frequency (%) of misconduct

Government-confirmed cases in the US

0.001 – 0.01 (Marshall 2000, Steneck 2006)

Retractions due to misconduct, in PubMed

0.02 - 0.2 (Claxton 2005)

Manipulated images submitted to The Journal of Cell Biology

1 (Steneck 2006)

Investigators disqualified by

FDA data audits (1977-1990)

2 (Glick 1992)

-Only misconduct that has been discovered,-Only misconduct that has been discovered,and (presumably) proven to be intentionaland (presumably) proven to be intentional

Ultimately, only scientists know about Ultimately, only scientists know about their own intentions! their own intentions!

NeverSometimesFrequently

Over the years, many surveys Over the years, many surveys have asked scientists directly…have asked scientists directly…

“Since entering medical school have you…?”

0 1-5 >5

“Have you participated in research involving […] during the last 10 years?”

“Fabricated data”

“Modified research or experimental results to improve the outcome”

“Failing to present data that contradict one's own previous research” “Indicate the number of

[…] members you have observed/experienced exhibiting […] within the last 5 years”

“Seriously misleading interpretation of results”

Yes No

QuestionQuestion Form of misconductForm of misconduct OutcomeOutcome

Results appeared inconclusive and dificult to compare

…different things, in different ways…

““Tricks” in the analysisTricks” in the analysis

• How many committed or observed X at least once• Only questions on fabrication, falsification, alteration and

QRP that distort scientific knowledge. No plagiarism, professional misconduct etc…

• Mixed questions were excluded

• No other measure of study quality (it’s controversial)– Included all eligible studies that specified their methods– Entered methodological factors in inverse variance weighted regression

ES=Log

)1( p

p

W=2

1

SE= )1( pn p

Question by question: effect size and weight

42 literature databases, 14 journals, 8 grey literature db, 2 internet scientific search engines, and references lists

The search…The search…"research misconduct" OR "research integrity" OR "research malpractice" OR "scientific fraud" OR "fabrication, falsification" OR "falsification, fabrication"

Potentially relevant studies obtained from electronic search (n=3276)

Studies retrieved for examination of full text (n=69)

Studies included in review (n=21)

Studies excluded for one of the following reasons (n=48):-Did not have any relevant or original data -Sample not exclusively composed of researchers-Misconduct not related to research (e.g. cheating on school projects)-Does not distinguish fabrication and falsification from other forms of misconduct not relevant to this review-Presents data only in format not usable in this review

Studies excluded because were not surveys on research misconduct (n=3207)

Studies included in meta-analysis (n=18)

Studies excluded from meta-analysis because did not meet quality criteria (n=3)

Characteristics of studiesCharacteristics of studies

• Conducted between 1986-2005

• USA (15), UK (3), multinational (2), and Australia (1)

• Medical/clinical (8), biomedical (6), multidisciplinary (6), economists (1)

• In total 85 questions:

– about fabrication, falsification, alteration, modification (meta-analysis)

– Questionable research practices (systematic review only)

(Full data set available soon in PLoS ONE)

Scientists who admit Scientists who admit fabrication, falsification, fabrication, falsification, or alteration of resultsor alteration of results

b= -0.14±0.05P=0.006

1.97% (N=7, 95%CI: 0.86-4.45) If only asked “fabrication, falsification”

1.06% (N=4, 95%CI: 0.31-3.51)

Scientists who know a Scientists who know a colleague who fabricated, colleague who fabricated, falsified, or altered resultsfalsified, or altered results

14.12% (N=12, 95% CI: 9.91-19.72) If only asked “fabrication, falsification”

12.34% (N=11, 95%CI: 8.43-17.71)

Questionable Research Practices

(e.g. “failing to publish data that contradicts one’s previous research”“dropping data points based on a gut feeling”)

Inverse variance-weighted regression

What influences admission rates?What influences admission rates?

Asking about self vs colleagues: b±SE P

-4.53±0.81 <0.0001

Handed-out surveys vs mailed: 1.17±0.4 0.0032

Using “fabrication” or “falsification” vs “alteration” or “modification”

-

+- -1.02±0.39 0.0086

82% of variance explained (N=15)

USA / other

Researcher / other

Biomedical / other

Social Sc. / other

Medical / other

Controlling for these factors, tested for differences between:

b=0.85±0.28

P= 0.0022

n.s.

Year

Martinson 2005 is Martinson 2005 is outstanding, as outstanding, as conservative!conservative!

Leave-one-out sensitivity analysisLeave-one-out sensitivity analysisScientists who admit Scientists who admit

fabrication, falsification, fabrication, falsification, or alteration of resultsor alteration of results

Scientists who know a Scientists who know a colleague who fabricated, colleague who fabricated, falsified, or altered resultsfalsified, or altered results

““Repairing Repairing misconduct” misconduct”

IDID N casesN cases Action takenAction taken %%

Tangney, 1987

78 Took some action to verify their suspicions of fraud or to remedy the situation

46

Rankin, 1997 31 (incl. Plag.)

In alleged cases of scientific misconduct a disciplinary action was taken by the dean

32.4

Some authority was involved in a disciplinary action 20.5

Ranstam, 2000

49 I interfered to prevent it from happening 28.6

I reported it to a relevant person or organization 22.4

Kattenbraker, 2007

33 Confronted individual 55.5

Reported to supervisor 36.4

Reported to Institutional Review Board 12.1

Discussed with colleagues 36.4

Titus, 2008 115 (incl. Plag.)

The suspected misconduct was reported by the survey respondent

24.4

The suspected misconduct was reported by someone else 33.3

Around half of recalled cases hadno action whatsoever taken against them

Summary of key findingsSummary of key findings

• Data fabrication, falsification and alteration was– admitted on average by around 2% (1% - 4%)

– directly observed by 14% (10% - 20%)

• Questionable Research Practices were– admitted on average by up to 34%

– directly observed by up to 72%

• Overall admission rates (self-/non-self) were higher in– Non-self reports, questions not using “fabrication” or

“falsification”, handed out questionnaires

– Medical/clinical and related research

Unlike surveys for other criminal behaviour,Scientists always lose by admitting misconduct

Unclear- Risk of multiple reporting- “Muhammad Ali” effect- Unaware of all cases- Unwilling to damage their field

Self-reports

Non-self-reports

Conservative

Regression analysis:Regression analysis:-Medical research not robust to all sensitivity-Differences in methodology masked most effects

-Non-significant effects not necessarily non-significant

How Reliable Are These Numbers?How Reliable Are These Numbers?

ConclusionsConclusions

• On average, 2% of scientists admitted misconduct, and 34% QRP– Actual frequencies probably higher– Probably vary depending on field and many

other factors, which meta-analysis currently cannot detect

• Future surveys might benefit by– Focusing on correlates of misconduct– Common methodologies

(tip of the iceberg)

(this too)