Joanne Yaffe, PhD, University of Utah Paul Montgomery, PhD, University of Oxford

How Empty Are Empty Reviews? The first report on the Empty Reviews Project sponsored by the Cochrane Opportunities Fund and an invitation to participate in

the development of reporting guidelines

Joanne Yaffe, PhD, University of UtahPaul Montgomery, PhD, University of Oxford

Sally Hopewell, PhD, UK Cochrane CentreLindsay Shepard, MSc, University of Utah

The Empty Reviews Project is funded by the Cochrane Opportunities Fund

Summary of Presentation

• The Problem of Empty Reviews• Current State of Cochrane Empty

Reviews and Apparent Direction of Travel• How this Project Aims to Proceed• Cochrane Contributor Inputs

The Cochrane Empty Reviews [email protected]

Introduction: Why study empty reviews?

• First, what do we mean by Empty Review?• An empty review is a systematic

review that reports no studies eligible for inclusion.


• Lack of high quality studies to address question posed. –Very narrow population–Very focused intervention–Outcomes that are not measured.

• Many plan empty reviews to justify new trials in response to funding policies.


How do systematic reviews come to be empty?

• Policy-makers and practitioners report frustration in locating a review only to find that it has no included studies and is of limited utility for clinical decisions.

• Should observations be made based on excluded studies to provide decision-makers with some guidance? (following Lang, Edwards & Flieszer 2007)


• The Cochrane Handbook (2009) does not address the reporting of empty reviews specifically, although it does address excluded studies in general.


“A Cochrane review includes a list of excluded studies, detailing any studies that a reader might plausibly expect to see among the included studies. This covers all studies that may on the surface appear to meet the eligibility criteria but on further inspection do not, and also those that do not meet all of the criteria but are well known and likely to be thought relevant by some readers. By listing such studies as excluded and giving the primary reason for exclusion, the review authors can show that consideration has been given to these studies. The list of excluded studies should be as brief as possible…”


Excluded Study Guidance in Cochrane Handbook (p. 154)

• Concern: Empty reviews may leave decision-makers dependent on weaker research which may be presented without sufficient discussion of limitations.

• Concern: Empty reviews may fail to provide any implications either for practice or research.


Lindsay

"Empty reviews may fail to provide ANY implications either for practice or research"

• Empty Reviews Project Aim: To develop guidelines, in consultation with stakeholders and the editors of the Cochrane Handbook, for authors of reviews who do not find studies eligible for inclusion.


Prevalence of Empty Reviews in The CDSR (as of 15 August 2010)

• Total reviews in The CDSR

N= 4320• Total empty reviews in The CDSR

N= 375 (8.7%)


Number of Empty Reviews across CRGs


0

5

10

15

20

25

30

35

40

Proportion of Empty Reviews across CRGs


0.0

5.0

10.0

15.0

20.0

25.0

30.0

Empty Reviews by Year of First Publication


1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 20100

10

20

30

40

50

60

70

Age of Empty Reviews in The CDSR

• Half of all empty reviews are 3 years or older, with some dating to 1995.• Mean Age = 3.3 years (SD 3.4)


Time since last update of Empty Reviews

• 48.3% of all empty reviews have been updated within the last 2 years, but 5.9% (21) have not received an update in 7 years or longer.• Mean Time since Update = 2.8 years

(SD 2.2)


How many excluded studies are listed in these empty reviews?

• 95 (25.3%) of empty reviews do not list excluded studies.• The number of excluded studies

ranges from 0 to 124. • Mean number of excluded studies =

9.6 (SD 14.5)The Cochrane Empty Reviews [email protected]

How many studies are listed as pending assessment?

• 360 (96.0%) of empty reviews do not list pending studies.• The number of pending studies

ranges from 0 to 9. • Mean number of pending studies =

0.1 (SD 0.7)The Cochrane Empty Reviews [email protected]

How many studies are listed as on-going?

• 338 (90.1%) of empty reviews do not list on-going studies.• The number of on-going studies

ranges from 0 to 4. • Mean number of on-going studies = 0.1

(SD 0.5)


How many reviews list no studies at all?

• 88 (23.5%) of empty reviews do not list any excluded, pending, or on-going studies.


So what?• So, what are the implications of all these empty

reviews? The lack of guidelines for reporting studies that have no

included studies may result in inconsistently reported reviews.

Policy-makers and practitioners might not rely on Cochrane reviews for decision-making.

More importantly, however, the lack of guidelines for authors of empty reviews may result in less than useful conclusions and further, may misstate results.


What do we mean by “less than useful conclusions”?


Less than useful conclusions.

What sorts of conclusions might be less than useful?

• Reference to single studies without appropriate caveats.• Unsupported statements.• No statements at all.


Review A : Implications for practiceSufficient evidence based on randomised controlled trials is lacking and therefore we can draw no reliable conclusions about the comparative effectiveness of medical and surgical interventions of [deleted to de-identify]. However, the studies identified, but excluded from this review, suggest there may be a considerable response to placebo treatment in this condition.

Review B: Implications for practiceOn the evidence currently available, [ ] cannot be recommended for the treatment of [ ].

Implications for research[ ] has shown promise in animal studies but there are no randomized double-blind placebo-controlled trials in people with [ ]. Such trials are needed.


What might be better ways to handle conclusions?

• Specific recommendations for design of trials.

• Appeal to related reviews, drawing analogy to present question, with caveats.

• Balanced discussion of weaker studies.


Review C: Implications for research 1. General

As with all similar reviews, public registration of a study before anyone is randomised would ensure that participants could be confident that people would know that the study had at least taken place. Compliance with CONSORT (Moher 2001), both on the part of authors and editors, would help to clarify methodology and many outcomes. Failure to comply with CONSORT guidelines results in loss of data and confusion in results, neither of which helps clinicians, patients or managers.

2. 2. Specific [ ] is a most distressing [ ] disorder that remains highly prevalent, both in the developed and developing world. This review highlights the need for well designed, conducted and reported clinical trials (Table 1) to address the claims of open studies as regards the effects of the [ ] group of drugs for [ ].


What do we hope to accomplish?

• Consistency in the way that excluded studies are addressed in the authors’ conclusions.

• Collaboration with other stakeholders towards development of guidelines for inclusion in The Cochrane Handbook.


Our Plan1. Complete our analysis of the 375 empty

reviews in The CDSR as of 15 August, 2010.

2. Solicit input from all interested parties via [email protected] and www.empty-reviews.org (coming soon!).

3. Develop specific recommendations around the writing, presentation, and persistence in updating empty reviews.


mailto:[email protected]

http://www.empty-reviews.org/

4. Present our findings and recommendations at a consensus meeting of key stakeholders in Spring, 2011, including Cochrane Handbook editors.

5. Use the results of the consensus meeting to refine our recommendations.

6. Present our findings to the Madrid Colloquium in October 2011, for further discussion.


The Cochrane Empty Reviews Project Funded by The Cochrane

Opportunities Fund

Paul Montgomery, PhD, University of Oxford, PISally Hopewell, PhD, UK Cochrane Centre, Co-PIJoanne Yaffe, PhD, University of Utah, Co-PILindsay Shepard, MSc, University of Utah, Research Associate

[email protected]://www.empty-reviews.org

[email protected]

mailto:[email protected]

http://www.empty-reviews.org/

Joanne Yaffe, PhD, University of Utah Paul Montgomery, PhD, University of Oxford

Documents

Transcript of Joanne Yaffe, PhD, University of Utah Paul Montgomery, PhD, University of Oxford