2014 CrossRef Annual Meeting Peer Review Panel: PRE: Securing Trust & Transparency in Peer Review

15
Securing Trust & Transparency in Peer Review http://www.pre-val.org @PeerReviewEval Adam Etkin, Founder and Managing Director November 2014

description

Adam Etkin presents the PRE system of evaluating peer review systems at the 2014 CrossRef annual meeting on a panel on innovations in peer review

Transcript of 2014 CrossRef Annual Meeting Peer Review Panel: PRE: Securing Trust & Transparency in Peer Review

Page 1: 2014 CrossRef Annual Meeting Peer Review Panel: PRE: Securing Trust & Transparency in Peer Review

Securing Trust & Transparency in Peer Review

http://www.pre-val.org

@PeerReviewEval

Adam Etkin, Founder and Managing Director

November 2014

Page 2: 2014 CrossRef Annual Meeting Peer Review Panel: PRE: Securing Trust & Transparency in Peer Review

A service that works with the publisher and journal to provide independent validation of the review process

A badge that publishers can display in various places – search results, article pages, article-level metrics – to signal to readers that quality peer review has been conducted

A window into a given journal’s peer review process accessible by end users

PRE-val answers the most basic, and important, question about scholarly works: “Has this article really been peer reviewed?”

PRE-val supports quality peer review.

What is PRE-val?

2

Page 3: 2014 CrossRef Annual Meeting Peer Review Panel: PRE: Securing Trust & Transparency in Peer Review

Why do we need PRE-val?

Traditional peer review & scholarly publishersseem to be under constant criticism.

Emergence of “predatory” publishers

High-profile cases of faulty research being published

Difficulty distinguishing peer-reviewed research from non-reviewed content in journals – publishing in a peer-reviewed journal does not equate with peer review for every article

“Publish then filter” and “Publishing is a button” attitude

3

Page 4: 2014 CrossRef Annual Meeting Peer Review Panel: PRE: Securing Trust & Transparency in Peer Review

Why do we need PRE-val?

4

Page 5: 2014 CrossRef Annual Meeting Peer Review Panel: PRE: Securing Trust & Transparency in Peer Review

Surveys show peer review is valued by researchers & authors.

Sources: Sense About Science; Taylor & Francis; CIBER Research; NPG/Palgrave Macmillan Author Insights survey

“The qualitative data also point to the fact that peer review is the central pillar of trust.”

University of Tennessee and CIBER Research Ltd, December 2013

Why do we need PRE-val?

5

Most (69%) researchers are satisfied with the current system of peer review but only a third think

that the current system is the best we can do

Most (69%) researchers are satisfied with the current system of peer review but only a third think

that the current system is the best we can do

Most (84%) believe that without peer review there would be no

control in scientific communication

Most (84%) believe that without peer review there would be no

control in scientific communication

78% of OA authors prefer traditional,

rigorous peer review

78% of OA authors prefer traditional,

rigorous peer review

Only 20% want basic check followed by post-

publication review

Almost all researchers (91%) believe that their last paper was

improved as a result of peer review

Almost all researchers (91%) believe that their last paper was

improved as a result of peer review

While many want a faster process with fewer rounds, the overwhelming majority (~70%) prefer to wait for

thorough review

While many want a faster process with fewer rounds, the overwhelming majority (~70%) prefer to wait for

thorough review

93% of science authors consider quality of

PEER REVIEW when deciding where to

publish

Page 6: 2014 CrossRef Annual Meeting Peer Review Panel: PRE: Securing Trust & Transparency in Peer Review

Less time to review more information

6

*B.-C. Björk, R. Annikki, and M. Lauri. Global annual volume of peer reviewed scholarly articles and the share available via different Open Access options*The National Science Board estimates the average annual growth of the indexes within the Web of Science to be 2.5% (See: Science and Engineering Indicators 2010, chapter 5, page 29)*The stm report: An overview of scientific and scholarly journal publishing (2009)*http://dan.corlan.net/cgi-bin/medline-trend?Q=

Total Peer Reviewed Journals

0

5,000

10,000

15,000

20,000

25,000

30,000

35,000

2006 2007 2008 2009 2010 2011 2012 2013

Year

Jou

rnal

s

28,000+ journals, and growing Total Peer Reviewed Articles Published

0

500,000

1,000,000

1,500,000

2,000,000

2,500,000

2006 2007 2008 2009 2010 2011 2012 2013

Year

Art

icle

s

2 million articles in 2013

More information than ever, less time to sort through it all.

Page 7: 2014 CrossRef Annual Meeting Peer Review Panel: PRE: Securing Trust & Transparency in Peer Review

Not just one “right” approach

The number of varied approaches to peer review is increasing, yet readers often assume it is a uniform practice

Journals with high standards have no clear way of routinely showing these standards at the article level

Trends toward “lite” peer review & critics are dominating the discussion, while quality has a harder time finding a voice

Peer review is not one thing, and it cannot be assumed.

This leads to confusion and questions about trust . . .

7

Page 8: 2014 CrossRef Annual Meeting Peer Review Panel: PRE: Securing Trust & Transparency in Peer Review

Measurements Abound

Page 9: 2014 CrossRef Annual Meeting Peer Review Panel: PRE: Securing Trust & Transparency in Peer Review

Metrics/Filters = Knowledge = Power

A Lesson From The Car Industry

Looking for a new car?• MPG• Size• Speed• Features• Safety• Cost

Evaluating a journal?• Impact Factor• Audience• Speed• Altmetrics• Peer Review (PRE) • Cost

Page 10: 2014 CrossRef Annual Meeting Peer Review Panel: PRE: Securing Trust & Transparency in Peer Review

10

Community

Different approaches, different services….one goal: Working together to educate and support

Page 11: 2014 CrossRef Annual Meeting Peer Review Panel: PRE: Securing Trust & Transparency in Peer Review

We believe in peer review

Recognizes journals with an editor-in-chief or other overseeing editor

PRE-val encourages journals to use quality reviewers

Provides transparency into iteration through review to improve the science and clarity of a paper

Helps promote use of best practices, which are markers of commitment to better peer review approaches

PRE-val creates incentives to use best practices in peer review.

11

Page 12: 2014 CrossRef Annual Meeting Peer Review Panel: PRE: Securing Trust & Transparency in Peer Review

Technical flow – Low/No work by publisher

12

First, we sit down together to learn about your process. PRE-val configuration takes into account your unique processes and preferences related to what you want to make public/transparent.

Publisher Manuscript Submission/Peer Review Tracking System

<metadata>PEER REVIEW</metadata>

PRE API

#1

Low/no development work on part of publisher or

manuscript submission/peer review system

Publisher places badge on - Journal article page - Search results - Aggregator sites - Article metrics - anywhere else a signal of peer review is important

Page 13: 2014 CrossRef Annual Meeting Peer Review Panel: PRE: Securing Trust & Transparency in Peer Review
Page 14: 2014 CrossRef Annual Meeting Peer Review Panel: PRE: Securing Trust & Transparency in Peer Review

Info about peer review process:•Type of Review•Rounds of review•Roles participating

Other measures of screening quality:•COPE member?•Plagiarism screening?•Retraction policy?

Optional info:•Reviewer comments•Reviewer names

14

Page 15: 2014 CrossRef Annual Meeting Peer Review Panel: PRE: Securing Trust & Transparency in Peer Review

Securing Trust & Transparency in Peer Review

http://www.pre-val.org

@PeerReviewEval

Adam Etkin, Founder and Managing Director

November 2014