Online - Exposing Scientific Peer Review (Oct08)

Peer Review in ScienceWhat is It, Does it Work, How Should It Evolve?

Alex J Mitchell, University of Leicester [email protected]

October 2008

Background:About Medical PublishingGutenburg publishing, increasing output

Slide Credit: Robert Gorter – Regional Manager Elsevier, 2008

http://www.americanscientist.org/my_amsci/restricted.aspx?act=pdf&id=3263000957901

Increase in Scientific Output

Publish Publish Publish!

Tony and Sam were pleased with their output for 2007

Author’s View of What’s Important

0102030405060708090

100Peer-reviewed

Refs' commentspublished

Referees identified

Public commentary oneprints

Post-publication publiccommentary

Ability to submitcomments

http://www.alpsp.org/pub5.ppt

Introducing Scientific Peer ReviewHistory, Definition, Types, Manuscript Processing

History of Peer Review

• Inquisition of the Holy Roman and Catholic Church”. Scholars’ works were examined for any hints of “heresy”.

• The first recorded academic peer review process was at The Royal Society in 1665 by the founding editor of Philosophical Transactions of the Royal Society, Henry Oldenburg, soon followed by “Medical Essays and Observations” published by the Royal Society of Edinburgh in 1731.

•Ray Spier (2002), "The history of the peer-review process", Trends in Biotechnology 20 (8), p. 357-358 [357].

• “Peer Review” of any type goes back to the 17th century and beyond; example “The

Definitions• Peer Review Process (defn from Wikipedia)

– Peer review (also known as refereeing) is the process of subjecting an author's scholarly work, research or ideas to the scrutiny of others who are experts in the same field.

– Peer review requires a willing and able community of experts who give impartial feedback, with no personal credit and no financial or other reward.

• Peer Review Journal– A peer-reviewed journal is one that has submitted most of its published

articles for review by experts who are not part of the editorial staff. The numbers and kinds of manuscripts sent for review, the number of reviewers, the reviewing procedures and the use made of the reviewers’

– (International Committee of Medical Journal Editors. Uniform Requirements for Manuscripts submitted to Biomedical Journals. 2001 http://www.icmje.org/)

Types of Peer ReviewInternal vs External

An internal review is conducted only by editorial staffAn external review is conducted by experts in the field

Blind vs OpenIn Blind Peer Review submitted manuscripts are sent outside of the journal’s

publishing or sponsoring organization for review by external reviewers whose identifies are hidden

In Open Peer Review reviewers disclose their identity. Often authors are encouraged to suggest possible reviewers who are may or may not be impartial

Examples of Problems with Peer Review• Famous papers that were published and did NOT get peer reviewed:

– Watson & Crick’s 1951 paper on the structure of DNA in Nature– Abdus Salam’s paper “Weak and electromagnetic interactions” (1968). Led to Nobel Prize– Alan Sokal’s “Transgressing the Boundaries...” in 1996 turned out to be a hoax. Now known as the

Sokal Affair.– Albert Einstein's revolutionary "Annus Mirabilis" papers in the 1905 issue of Annalen der Physik were

only reviewed by the editor • Famous papers that were published and passed peer review that later proved to be fraudulent:

– Jan Hendrik Schon (Bell Labs) submitted and passed peer review 15 papers published in Science and Nature (1998-2001) found to be fraudulent.

– Igor and Grichka Bogdanov 1999 & 2002 published papers in theoretical physics believed by many to be jargon-rich nonsense.

• Famous papers that got rejected that later turned out to be seminal works:– Krebs & Johnson’s 1937 paper on the role of citric acid on metabolism was rejected by Nature as being

of “insufficient importance”, was eventually published in the Dutch journal Enzymologia. This discovery, now known as the Krebs Cycle, was recognized with a Nobel prize in 1953.

– Black & Scholes 1973 paper on “the pricing of options and corporate liabilities”, rejected many times, was eventually published at the intersession of Merton Miller to get it accepted by the Journal of Political Economy. This work led to the Nobel Prize.

Credit: Peggy Dominy & Jay Bhatt

Does it Work? The Issue In a Nutshell• “Stand at the top of the stairs with a pile of papers and throw them

down the stairs. Those that reach the bottom are published.”

=> It is not clear to what extent peer review improves the submitted product

• “Sort the papers into two piles: those to be published and those to be rejected. Then swap them over!”

=> Peer review is often haphazard and unmonitored

Adapted from Trish Groves: BMJ What Do We Know About Peer Review

Typical Journal Statistics (eg BMJ)• 6000-7000 research papers received• About 5-7% accepted• 1000 rejected by one editor within 48 hours

– further 3000 rejected with second editor– within one week of submission 3000 read by senior editor;

further 1500 rejected– 1500 sent to two reviewers; then 500 more rejected– 500 reach weekly manuscript meeting (with Editor, a clinicians

and a statistician)– 350 research articles accepted, usually after revision

• See Over for Detailed Figure

BMJ Manuscript Processing

5000 scanned by 1 editor

2500 sentfor review

1000 Immediately

rejected

6000 received per year

Poorly writtenObvious flawsToo Obscure

Hand writtenSilly mistakesWrong journal

100 accepted

2500Rejected by

editor

1000 Rejected by reviewer /

editor

Methodological concerns

Not interesting

400 rejected

1500 sent to hanging

committees

500 discussed

100 accepted400 rejected500 discussed

100 accepted400 rejected500 discussed

300 accepted per year(6 per week + 2 short

reports)

Acceptance Rate = 1 in 20

Functions/Responsibilities of Peer Review

1. Filtering out incorrect, inadequate & fraudulent work

2. Improving the accuracy and clarity of work that warrants acceptance

3. Helping journals deal with high volumes

4. Helping journals deal with multiple publication

These Are NOT Functions of Peer Review

• Deciding whether the paper should be accepted– This is the role of the editor

• Improving the spelling and grammar– This is the role of the copy-editor

• Improving on the study design– This is the role of the author

• Deciding upon the author order– This is the role of the authors

• Disseminating the reviewed paper– This is not allowed unless the paper is officially in print

Practicalities of Peer ReviewFilters, Publication Cycle, Responsibilities, Tips

Overview of Peer Review

Grading Accept ReviseReject

Peer Review Filter

Published Article

Send elsewhere

Reject

Qualitative Grade

Credit: Bradley Hemminger

Editorial Filter

Quantitative Grade

Outcome

Comments to Author

Score (1-10)

Author Filter

Article Appraisal

Where to Submit

Conventional Publication Cycle – 12mo

Step1

Paper Completed

Step 2

Present to colleagues (informal PR)

Step 3

Present at conference

Step 4

Submit to journal

Step 6

Peer

Review 1

Step 7

Revision 1

Step 8

Peer Review 2

Step 10

Proof

T0 T+1mo

Step 12

Readers letters (public PR)

T+3mo T+6mo

Step 11

Restricted Publication

Step 9

Revision 2

Step 5

Editor Agrees to Send for Review

T+7mo T+10mo T+12mo T+14mo

T+15mo T+18mo T+24mo

Questions before starting to Review• Expertise:

– Do I have expertise in the content or methods, or a valuable perspective on the issue?

• Potential conflicts : – Do I have conflicts of interest that preclude fair and balanced judgments?– Do I stand to gain, either financially or personally, from reviewing this

particular manuscript?– Will I be able to hold the main information that I gain from reviewing this

manuscript confidential until publication?

• Ability to meet deadline– Do I have the time to devote to this review and complete it by the date the

editors requested?

Peer Review: Integral to Science and Indispensable to Annals 1038 16 December 2003 Annals of Internal Medicine Volume 139 • Number 12

Questions whilst completing the Review• Does the review address the relevance of the topic to readers?• Does the review address the manuscript’s importance and novelty and say what

it adds to existing knowledge?• Does the review address the validity of the research, pointing out major strengths

and weaknesses of the methods?• Does the review address the clarity of presentation?• Does the review address important missing and/or inaccurate• information?• Does the review address the generalizability of findings?• Does the review address the interpretation of results and stated conclusions?• Does the review address whether the authors noted and discussed important

limitations?• Does the review cite specifics to support criticisms?• Does the review offer suggestions for improvement?• Does the review keep nitpicking to a minimum?• Is the review’s tone balanced?• Did I declare potential conflicts of interest?

Peer Review: Integral to Science and Indispensable to Annals 1038 16 December 2003 Annals of Internal Medicine Volume 139 • Number 12

Editor’s Dilemma• What Happens When Referees Disagree:

– (a) the Editor must decide– (b) Option to Use more additional reviews as a tie-breaker– (c) invite authors to reply to a referee's criticisms– (d) an editor may convey communications back and forth

between authors and a referee

– Usually however the editor will reject the paper if there is 1x negative review unless there is a special reason not to do so. This process is not open, and such decisions are often unexplained to authors

Tips for Reviewers: 1• Be courteous and constructive• Your role is advising not deciding• Try to suggest improvement no matter what the outcome• Maintain confidentiality• Don’t review work for those you know well• Complete reviews promptly, typically within 4 weeks• Spend at least 1 hour on the review• Search for related (esp recent work)• Write as you would like to be written to

Tips for Reviewers 2 - Key Questions• Is the research question Appropriate?• Was the question answered?• Were the methods appropriate?• What must be improved?• What could be improved?• What were the strengths?• Was all relevant literature considered?• What will readers think• => Would I object if my review was published

Problems with Peer ReviewExplicit Rules , Reliability, Bias, Blinding, Plagiarism Detection, Delays

Rules of the Game are Not ExplicitPeters and Ceci (1982)Resubmitted 12 altered articles to psychology journals that

had already published them but changed title/abstract/introduction/authors’ name/name of institution

• 3 articles recognised as resubmissions• One accepted• 8 rejected on methdological grounds!

Peters, D.C., & Ceci, S.J. (1982). Peer review practices of psychological journals: The fate of published articles, submitted again. The Behavioral and Brain Sciences, 5, 187-255.

What’s Wrong with Peer Review?• For Authors, Peer review:

– Usually Unreliable (Idiosyncratic)– Often Unfair (many biases)– Has no gold standard (Unstandarized)– Often provides qualitative comments only

• For Journals, Peer review:– Stifles innovation => encourages group think– Rewards the prominent– Causes unnecessary delay in publication– Is very expensive– Does not detect fraud, duplicate publication etc

Juan Miguel Campanario, "Rejecting Nobel class articles and resisting Nobel class discoveries", cited in Nature, 16 October 2003, Vol 425, Issue 6959, p.645Sophie Petit-Zeman, "Trial by peers comes up short" (2003) The Guardian, Thursday January 16, 2003

Inter-Rater Reliability Issues - 1• Locke reported inter-observer values ranging from 0.11 to 0.49 for agreement

between a number of reviewers making recommendations on a consecutive series of manuscripts submitted to BMJ

• Ingelfinger reported Rates of agreement only “moderately better than chance” (Kappa = 0.26) with agreement greater for rejection than acceptance

• Strayhorn and colleagues reported a value of 0.12 (poor agreement) for 268 manuscripts submitted to the JAACAP (Strayhorn et al., 1993)

• low levels of agreement were reported by Scharschmidt et al. for papers submitted to the Journal of Clinical Investigation (Scharschmidt et al., 1994

Two reviewers are not enough• Fletcher and Fletcher 1999 - need at least six reviewers, all favouring rejection or

acceptance, to yield a stats significant conclusion (p<0.05)

Ingelfinger FJ. Peer review in biomedical publication. American Journal of Medicine 1974; 56:686-692.. Locke S. A difficult balance: editorial peer review in medicine. London: Nuffield Provincial Hospitals Trust; 1985.Strayhorn J Jr, McDermott JF Jr, Tanguay P. An intervention to improve the reliability of manuscript reviews for the Journal of the American Academy of Child and Adolescent Psychiatry. Am J Psychiatry 1993; 150: 947–52.Scharschmidt BF, DeAmicis A, Bacchetti P, Held MJ. Chance, concurrence and clustering: analysis of reviewers' recommendations on 1000 submissions to the Journal of Clinical Investigation. J Clin Invest 1994; 93: 1877–80.

Inter-Rater Reliability Issues - 2• Linkov F et al Quality Control of Epidemiological Lectures Online:

Scientific Evaluation of Peer Review Croat Med J. 2007;48:249-55

Is agreement between reviewers any greater than would be expected by chance alone? Brain 2000

• We studied two journals in which manuscripts were routinely assessed by two reviewers, and two conferences in which abstracts were routinely scored by multiple reviewers.

• Agreement between the reviewers as to whether manuscripts should be accepted, revised or rejected was not significantly greater than that expected by chance.

Rothwell PM & Martyn CN Reproducibility of peer review in clinical neuroscience Is agreement between reviewers any greater than would be expected by chance alone? Brain, Vol. 123, No. 9, 1964-1969, 2000

• Editors were very much more likely to publish papers when both reviewers recommended acceptance than when they disagreed or recommended rejection

• There was little or no agreement between the reviewers as to the priority

Reliability of Editors' Subjective Quality Ratings of Peer Reviews of Manuscripts

• Objective.— Whether editors' quality ratings of peer reviewers are reliable and how they compare with other performance measures.

• Design.— A 3.5-year prospective observational study. • Participants.— All editors and peer reviewers who reviewed at least 3 manuscripts. • Main Outcome Measures.— Reviewer quality ratings, individual reviewer rate of

recommendation for acceptance, congruence between reviewer recommendation and editorial decision (decision congruence), and accuracy in reporting flaws in a masked test manuscript.

• Interventions.— Editors rated the quality of each review on a subjective 1 to 5 scale. • Results.— A total of 4161 reviews of 973 manuscripts by 395 reviewers were studied.

The within-reviewer intraclass correlation was 0.44 (P<.001), indicating that 20% of the variance seen in the review ratings was attributable to the reviewer. Intraclass correlations for editor and manuscript were only 0.24 and 0.12, respectively. Reviewer average quality ratings correlated poorly with the rate of recommendation for acceptance (R=-0.34) and congruence with editorial decision (R=0.26). Highly rated reviewers reported twice as many flaws as poorly rated reviewers.

• Conclusions.— Subjective editor ratings of individual reviewers were moderately reliable and correlated with reviewer ability to report manuscript flaws.

Callaham ML et al Reliability of Editors' Subjective Quality Ratings of Peer Reviews of Manuscripts JAMA. 1998;280:229-231.

BiasAuthor-related• Prestige (author/institution)• Gender• Where they live and work

Paper-related• Positive results• English language

Bias from Hidden Reviewers

AuthorAuthor

Friend

Unknown

Unknown

Enemy

Revision

Rejection

Enemy

+

+

--

-

Acceptance

See Maddox J. Conflicts of interest declared [news]. Nature 1992; 360: 205; Locke S. Fraud in medicine [editorial]. Br Med J 1988; 296: 376–7.

Open & Blind Review & Submission• Does revealing reviewers’ identity influence outcome?

– Open vs blind peer review

• Does revealing Authors’ identity influence outcome?– Open vs blind submission

• Key Studies

“Effect of Blinding and Unmasking on the Quality of Peer Review A Randomized Trial JAMA”

Design and Setting.— Randomized trial of 527 consecutive manuscripts submitted to BMJ, which were randomized and each sent to 2 peer reviewers.

• Interventions.— Manuscripts were randomized as to whether the reviewers were unmasked, masked, or uninformed that a study was taking place. Two reviewers for each manuscript were randomized to receive either a blinded or an unblinded version.

• Results.— Of the 527 manuscripts entered into the study, 467 (89%) were successfully randomized and followed up. The mean total quality score was 2.87. There was little or no difference in review quality between the masked and unmasked groups (scores of 2.82 and 2.96, respectively) and between the blinded and unblinded groups (scores of 2.87 and 2.90, respectively). There was no apparent Hawthorne effect. There was also no significant difference between groups in the recommendations regarding publication or time taken to review.

• Conclusions.— Blinding and unmasking made no editorially significant difference to review quality, reviewers' recommendations, or time taken to review.

Van Rooyen et al Effect of Blinding and Unmasking on the Quality of Peer Review A Randomized Trial JAMA. 1998;280:234-237.

Open vs Blind Submission

“Effect on the quality of peer review of blinding peer reviewers and asking them to sign their reports”

• Objective.— To evaluate the effect on the quality of peer review of blinding reviewers to the authors' identities and requiring reviewers to sign their reports.

• Design.— Randomized controlled trial. • Setting.— A general medical journal. • Participants.— A total of 420 reviewers from the journal's database. • Intervention.— We modified a paper accepted for publication introducing 8 areas of weakness.

Reviewers were randomly allocated to 5 groups. Groups 1 and 2 received manuscripts from which the authors' names and affiliations had been removed, while groups 3 and 4 were aware of the authors' identities. Groups 1 and 3 were asked to sign their reports, while groups 2 and 4 were asked to return their reports unsigned. The fifth group was sent the paper in the usual manner of the journal, with authors' identities revealed and a request to comment anonymously. Group 5 differed from group 4 only in that its members were unaware that they were taking part in a study.

• Main Outcome Measure.— The number of weaknesses in the paper that were commented on by the reviewers.

• Results.— Reports were received from 221 reviewers (53%). The mean number of weaknesses commented on was 2 (1.7, 2.1, 1.8, and 1.9 for groups 1, 2, 3, and 4 and 5 combined, respectively). There were no statistically significant differences between groups in their performance. Reviewers who were blinded to authors' identities were less likely to recommend rejection than those who were aware of the authors' identities (odds ratio, 0.5; 95% confidence interval, 0.3-1.0).

• Conclusions.— Neither blinding reviewers to the authors and origin of the paper nor requiring them to sign their reports had any effect on rate of detection of errors. Such measures are unlikely to improve the quality of peer review reports

Godlee et al JAMA. 1998;280:237-240.


“The effects of blinding on the quality of peer review. A randomized trial”• Peer reviewers are blinded sometimes to authors' and institutions' names, but the

effects of blinding on review quality are not known.

• We, therefore, conducted a randomized trial of blinded peer review. Each of 127 consecutive manuscripts of original research that were submitted to the Journal of General Internal Medicine were sent to two external reviewers, one of whom was randomly selected to receive a manuscript with the authors' and institutions' names removed.

• Reviewers were asked, but not required, to sign their reviews.• Blinding was successful for 73% of reviewers.

• Quality of reviews was higher for the blinded manuscripts (3.5 vs 3.1 on a 5-point scale). Forty-three percent of reviewers signed their reviews, and blinding did not affect the proportion who signed. There was no association between signing and quality. Our study shows that, in our setting, blinding improves the quality of reviews and that research on the effects of peer review is possible.

McNutt et al JAMA Vol. 263 No. 10, March 9, 1990


“Does Masking Author Identity Improve Peer Review Quality? A Randomized Controlled Trial”• Objectives.— To determine whether masking reviewers to author identity is generally associated

with higher quality of review at biomedical journals, and to determine the success of routine masking techniques.

• Interventions.— Two peers reviewed each manuscript. In one study arm, both peer reviewers received the manuscript according to usual masking practice. In the other arm, one reviewer was randomized to receive a manuscript with author identity masked, and the other reviewer received an unmasked manuscript.

• Main Outcome Measure.— Review quality on a 5-point Likert scale as judged by manuscript author and editor. A difference of 0.5 or greater was considered important.

• Results.— A total of 118 manuscripts were randomized, 26 to usual practice and 92 to intervention. In the intervention arm, editor quality assessment was complete for 77 (84%) of 92 manuscripts. Author quality assessment was complete on 40 (54%) of 74 manuscripts. Authors and editors perceived no significant difference in quality between masked (mean difference, 0.1; 95% confidence interval [CI], -0.2 to 0.4) and unmasked (mean difference, -0.1; 95% CI, -0.5 to 0.4) reviews. We also found no difference in the degree to which the review influenced the editorial decision (mean difference, -0.1; 95% CI,-0.3 to 0.3). Masking was often unsuccessful (overall, 68% successfully masked; 95% CI, 58%-77%), although 1 journal had significantly better masking success than others (90% successfully masked; 95% CI, 73%-98%). Manuscripts by generally known authors were less likely to be successfully masked (odds ratio, 0.3; 95% CI, 0.1-0.8). When analysis was restricted to manuscripts that were successfully masked, review quality as assessed by editors and authors still did not differ.

• Conclusions.— Masking reviewers to author identity as commonly practiced does not improve quality of reviews. Since manuscripts of well-known authors are more difficult to mask, and those manuscripts may be more likely to benefit from masking, the inability to mask reviewers to the identity of well-known authors may have contributed to the lack of effect.

Justice et al. JAMA. 1998;280:240-242.


“Differences in Review Quality and Recommendations for Publication Between Peer Reviewers Suggested by Authors/Editors"

• Design, Setting, and Participants Observational study of original research papers sent for external review at 10 biomedical journals. Editors were instructed to make decisions about their choice of reviewers in their usual manner. Journal administrators then requested additional reviews from the author's list of suggestions according to a strict protocol.

• Main Outcome Measure Review quality using the Review Quality Instrument and the proportion of reviewers recommending acceptance (including minor revision), revision, or rejection.

• Results There were 788 reviews for 329 manuscripts. Review quality (mean difference in Review Quality Instrument score, –0.05; P = .27) did not differ significantly between author- and editor-suggested reviewers. The author-suggested reviewers were more likely to recommend acceptance (odds ratio, 1.64; 95% confidence interval, 1.02-2.66) or revise (odds ratio, 2.66; 95% confidence interval, 1.43-4.97). This difference was larger in the open reviews of BMJ than among the blinded reviews of other journals for acceptance(P = .02). Where author- and editor-suggested reviewers differed in their recommendations, the final editorial decision to accept or reject a study was evenly balanced (50.9% of decisions consistent with the preferences of the author-suggested reviewers).

• Conclusions Author- and editor-suggested reviewers did not differ in the quality of their reviews, but author-suggested reviewers tended to make more favorablerecommendations for publication.

Schroter JAMA 2006;295:314-317; see also Scharschmidt et al J Clin Invest 1994; 93: 1877–80.

Choice vs No-Choice Reviewer

Gutenburg publishing, increasing output

COPE Studies• Deliberately inserted 8 errors(method, analysis and

interpretation) into an accepted paper

• sent it to 400 reviewers - 221 responded

• mean number of weaknesses found was 2

• only 10% identified 4 or more

• 16% didn’t detect any •Godlee,F et al JAMA,1998,280,237

“Open peer review: a randomised controlled trial”Br Journal Psychiatry 2000

• 408 manuscripts assigned to reviewers who agreed were randomised to signed or unsigned groups.

• 245 reviewers (76%) agreed to sign their name.

• Signed reviews were of 5% higher quality, more courteous and took longer to complete than unsigned reviews.

• They were more likely to recommend publication.

Walsh et al (2000) Br Journal Psychiatry 176(1)47-51

van Rooyen BMJ 1999;318:23-27• 125 eligible papers were sent to two reviewers who were

randomised to have their identity revealed to the authors or to remain anonymous.

• Identified were 12% more likely than anonymous (35% v 23%) to decline to review the paper.

• There was no significant difference in quality

• No significant difference in the recommendation regarding publication or time taken to review the paper.

Problems: Plagarism & Duplicates

• A poll of 3,247 scientists funded by the U.S. National Institutes of Health found 0.3% admitted faking data, 1.4% admitted plagiarism, and 4.7% admitted to autoplagiarism (republishing).[Weiss]

• Note: Reviewers generally lack access to full raw data!!

• Weiss, Rick. 2005. Many scientists admit to misconduct: Degrees of deception vary in poll. Washington Post. June 9, 2005. page A03.[1]

Problems: Delays!!

“There was some delay in the peer review due to a backlog in the in-tray”

Credit: [email protected]

Speed of Publication

Credit: Fiona Godlee

Evidence behind Peer ReviewWhat does the evidence suggest….who judges the judges?

“Effects of Editorial Peer Review: A Systematic Review”• 9 studies considered the effects of concealing reviewer/author identity.

– 4 studies suggested that concealing reviewer or author identity affected review quality (mostly positively); however, methodological limitations make their findings ambiguous.

• One study suggested that a statistical checklist can improve report quality, but another failed to find an effect of publishing another checklist.

Jefferson T et al JAMA. 2002;287:2784-2786

• 2 studies of how journals communicate with reviewers did not demonstrate any effect on review quality.

• 1 study failed to show reviewer bias.• 1 nonrandomized study compared the quality of articles

published in peer-reviewed vs other journals• Two studies showed that editorial processes make articles

more readable and improve the quality of reporting

Future of Peer ReviewOpen publication, training, what makes a good reviewer?

Open Publication Cycle – 12 monthsStep1

Paper Completed

Step 2

Present to colleagues (informal PR)

Step 3

Present at conference

Step 4

Submit to online journal

Step 7 Peer Review 1

Step 6 Public comments

Step 8

Revision

T0 T+1mo

Step 11

Readers letters (post-publication)

T+3mo T+5mo

Step 10

Open Access Publication

Step 9

Online proof

Step 5

Editor Publishes “discussion paper”

T+6mo T+7mo T+9mo

T+11mo T+12mo

New Models: Case 1

• All submitted articles within scope are immediately posted on the Web for a 90 day discussion period

• At end of “review” period, authors given option to revise; revised article sent out for “pass-fail” review”

• If “pass,” article is published

New Models: Case 2

• Authors select reviewers from among BD editorial board members• Reviews published alongside author’s responses as part of article• Three reviews required

New Models: Case 3• Pre-publication review focuses

on technical rather than subjective issues

• All published papers made available for community-based open peer review including online annotation, discussion, and rating

• Managing Editor, Chris Surridge

Discussion Forum (Pub. Stage 1) + Journal (Pub. Stage 2)Open Access Publishing

Publish without a

Publisher?

Improving Peer Review by Training?

Effects of training on quality of peer review: randomised controlled trial

• Reviewers in the self taught group scored higher in review quality after training than did the control group, but the difference was not of editorial significance and was not maintained in the long term.

– Both intervention groups identified significantly more major errors after training than did the control group (3.14 and 2.96 v 2.13; P < 0.001), and this remained significant after the reviewers' performance at baseline assessment was taken into account.

– The evidence for benefit of training was no longer apparent on further testing six months after the interventions. Training increased likelihood of recommending rejection (92% and 84% v 76%; P = 0.002).

• Short training packages have only a slight impact on the quality of peer review.

• http://resources.bmj.com/bmj/reviewers/training-materials

Schroter et al BMJ 2004;328:673

What errors do peer reviewers detect, and does training improve their ability to detect them? • Design 607 peer reviewers at the BMJ were randomized to two intervention

groups receiving different types of training (face-to-face training or a self-taught package) and a control group. Each reviewer was sent the same three test papers over the study period, each of which had nine major and five minor methodological errors inserted.

• Main outcome measures The quality of review, assessed using a validated instrument, and the number and type of errors detected before and after training.

• Results The number of major errors detected varied over the three papers. The interventions had small effects. At baseline (Paper 1) reviewers found an average of 2.58 of the nine major errors, with no notable difference between the groups. The mean number of errors reported was similar for the second and third papers, 2.71 and 3.0, respectively. Biased randomization was the error detected most frequently in all three papers, with over 60% of reviewers rejecting the papers identifying this error. Reviewers who did not reject the papers found fewer errors and the proportion finding biased randomization was less than 40% for each paper.

• Conclusions Editors should not assume that reviewers will detect most majorerrors, particularly those concerned with the context of study. Short training packages have only a slight impact on improving error detection

J R Soc Med 2008;101:507-514

What Makes a Good Reviewer?• A reviewer was less than 40 years old

• From a top academic institution

• Well known to the editor choosing the reviewer

• Author blinded to the identity of the manuscript's authors

• Then the probability of good review was 87% vs 7%

Evans AT, McNutt RA, Fletcher SW, Fletcher RH. The characteristics of peer reviewers who produce good-quality reviews. J Gen Intern Med. 1993 Aug;8(8):422-8

Why Credit for Reviewing?• See:

• Leanne Tite, Sara Schroter Why do peer reviewers decline to review? A survey Journal of Epidemiology and Community Health 2007;61:9-12; doi:10.1136/jech

Richard Horton (editor The Lancet):

• "The mistake, of course, is to have thought that peer review was any more than a crude means of discovering the acceptability — not the validity — of a new finding. Editors and scientists alike insist on the pivotal importance of peer review. We portray peer review to the public as a quasi-sacred process that helps to make science our most objective truth teller. But we know that the system of peer review is biased, unjust, unaccountable, incomplete, easily fixed, often insulting, usually ignorant, occasionally foolish, and frequently wrong."

eMJA: Horton, Genetically modified food: consternation, confusion, and crack-up

Overview of Changes

DeclaredAnonymousDeclaredFuture

DeclaredPartial Anony.Partially declared

Current

DeclaredDeclaredHiddenPast

Paper in printAuthor (Draft) identity

Peer Identity

10 Suggestions for Better Peer Review1. Make Peer Review part of training (with supervision)2. Disclose the identifies of the reviewers3. Hide the identities of the authors4. Peer review with a minimum of 3 reviewers5. Have a quantitative and qualitative element6. Allow anyone to comment whilst the article is “online only”7. Score every peer review and avoid use of low scorers8. Require reviewers to disclose absence of bias & require authors to

submit full datasets9. Show the authors the full review on request10. Publish the peer reviews for every paper (accepted or rejected)

online

Online - Exposing Scientific Peer Review (Oct08)

Education

Transcript of Online - Exposing Scientific Peer Review (Oct08)