Writing Reviews Worth Reading

Tina Jones

Gavin Leslie

Andrea Marshall

The Traditional Literature review versus a Systematic Review

Dr Tina JonesManager, Australian Centre for Evidence

Based Clinical Practice, FMCSenior Lecturer, Flinders University

An Evidence Based Approach to Clinical Practice

• The Evidence Based Practice (EBP) framework emerged in the early 1970s as a means of improving clinical practice

• Shifted decision-making from a culture of delivering care based on tradition, intuition and authority

• To a framework where clinical decisions are based on best available evidence

Where do nurses find the evidence?

• 38.7% nursing journals (Estabrooks 1998)

• > 400 nursing journals listed

• Access barriers– Lack of time and easy access– Lack of evidence related to nursing– Lack confidence in ability to locate,

understand and properly evaluate research – Lack of organisational infrastructure support

(Thompson et al 2001, Nagy et al 2001)

A Traditional Literature Review

• Narrative in style

• Written by clinical experts or individuals who have read widely on a particular topic

• Provide a qualitative summary of the literature

• Use an informal and subjective approach to identifying and appraising the literature

Davies H. & Leslie G. 2006 Maintaining the CRRT circuit: non-anticoagulant alternatives, ACC; 19(4):133-138

Limitations of the Traditional Review

• Traditional reviews have been criticised as haphazard and biased, subject to the idiosyncratic impressions of the individual reviewer (Mulrow 1987)

• Potential sources of bias– Lack of explicit inclusion or exclusion criteria for the

studies cited– Reviewer bias towards particular views/practices– No mention of methods used to assess studies, or

their variable quality– Lack rigor in both synthesis of findings and data

extraction

Limitations of the Traditional ReviewMulrow (1987) from June 1985-1986 identified 50 reviews in 4 major medical journals (Ann of Int Med, Arch of Int Med, JAMA, NEJM)

• Criteria:– Purpose of review, search methods and sources, reasons for

inclusion & exclusions, methodological assessment, explication of data inconsistencies, integration of findings, summary of pertinent findings, directives for new research

• ‘Current medical reviews do not routinely use (such) scientific methods to identify, assess, and synthesise information’ (Mulrow 1987)

• The more authoritative the expert writing the review, the lower the quality of the review (Oxman & Guyatt 1995)

Systematic Reviews

A review of the relevant literature on a focused clinical question that employs explicit methods to minimise bias in:

• the conduct of the literature search • the appraisal of the individual studies• the methods for pooling and summarising

data

Gardner A et al 2005 Best practice in stabilisation of oral endotracheal tubes: a systematic review ACC; 18(4):158- 165

Purpose of Systematic Reviews

“The purpose of a systematic literature review is to evaluate and interpret all available research evidence relevant to a particular question”

(NHMRC, 2000)

Why do we need Systematic Reviews?

• Invaluable to health care providers, researchers and policy makers

• Inundated with unmanageable amounts of information

• Powerful means of integrating the best research evidence

• Rational basis for health care decisions• Potential to improve healthcare outcomes

Why do we need Systematic Reviews?

• To avoid the personal bias of an author

• ‘How to live longer and feel better’ – Linus Pauling (1986)– large doses of vitamin C prevent colds

• Systematic review showed Vitamin C in doses as high as 1 g/day for several winter months, had no consistent beneficial effect on incidence of the common cold (Douglas et al 2003)

Why Systematic Reviews Matter…

Why Systematic Reviews Matter…

Why Systematic Reviews Matter…

• Allow data to be ‘pooled’ and summarised, increasing statistical ‘power’

• A meta-analysis is a statistical technique for combining the findings of two or more studies in order to provide a single quantitative estimate of the overall treatment effect

The Power of Systematic Reviews

• 1972 the first RCT of prenatal corticosteroids to women at risk of preterm birth

• Within 10 years, 7 more trials had been completed, all showing benefit

• Because no-one had ‘collated’ these findings until 1989, there was limited use of prenatal corticosteroids until early 1990s

The Power of Meta-analysis

NHMRC Levels of Evidence

Level I Systematic Review of Level II studiesLevel II Randomised Controlled TrialLevel III-1 Pseudo Randomised Controlled TrialLevel III-2 Comparative Study with concurrent

controlsLevel III-3 Comparative Study without concurrent

controlsLevel IV Case Series

(NHMRC, 1999)

Who conducts systematic reviews & where to find them?

Basic steps in systematic review process – question identification, literature review,

and compilation of data.

Assoc Prof Gavin D Leslie

Editor - Australian Critical Care

Who conducts systematic reviews & where to find them?

• Anyone can conduct a review

• Numerous EBP “centres” specialising in reviews - e.g.– Cochrane Collaboration (medical)– Joanna Briggs institute (nursing, midwifery)– Pedro (physiotherapy)

• Remember - not all reviews are the same!

Cochrane Collaborationhttp://www.cochrane.org.au/

• Best known (est 1996) international collaboration

• Over 2000 reviews available

• Predominantly based on RCT

• Medically orientated

• Strict criteria and process

Joanna Briggs Institutehttp://www.joannabriggs.edu.au/about/home.php

• Australian based international group

• Nursing & midwifery focussed

• different criteria and approach to Cochrane, although principles are the same

• Well known for “Practice Summaries”

Where to find them?

• Numerous search routes (web based)-

• original site - limited to own reviews

• Specialised search facilities e.g.– TRIP

http://www.tripdatabase.com/index.html

• Generic search facilities e.g.– OVID

• Your librarian!

Where to find them?

• EBP “promoters”– NICS– http://wherestheevidence.nicsl.com.au/asp/

index.asp

– ACEBP– http://www.acebcp.org.au/intro.htm

– scharr - netting the evidence

– http://www.shef.ac.uk/scharr/ir/netting/

Basic steps in systematic review process – question identification.

• A clearly identified question - PICO– Patient– Intervention– Comparison– Outcome

Framing the Question

patients /problem

intervention comparison outcome

All adultpatients whorequire venouspuncture fordrawing bloodor IVcannulation.

Insertion site skinpreparation

Saline vs alcowipe vs iodine vschlorhexidine

Minimal risk ofsitecontamination.

Question

What is the preferred skin site preparation for venous cannulation in adult patients?

Basic steps in systematic review process – literature review

• Search Strategy• Use a thorough search strategy

– Describe your search strategy, list key words – List databases accessed, reference lists, other lit

sources, language

• Eliminate inclusion bias– Devise a selection process prospectively with

inclusion and exclusion criteria– Study designs, levels of evidence (RCTs etc)

Basic steps in systematic review process –literature review

• Process of Critical Appraisal– Data Extraction & Synthesis. Errors minimised (use

checklists), tools identified (objective criteria or measures), use 2 independent reviewers (and a third arbiter)

• Provide details of all studies reviewed, both included and not included– Important in really deciding the integrity of the

review.

– Provided in full review - not usually in “article” size version.

Basic steps in systematic review process – compilation of data

• Evaluation of studies cumulatively using statistical assessment.– Forrest plots, risk ratios,

CIs

New developments in systematic reviews

Andrea MarshallSesqui Senior Lecturer Critical Care

The University of Sydney

Associate Editor, Australian Critical Care

Advantages of systematic reviews

• Limit bias• Conclusions more reliable and accurate • Assimilate large amounts of information • Formally compared studies

– generalisability of findings and consistency of results

• Identify inconsistency in results and generate new hypotheses

• Increase the precision of the overall result

Limitations of Systematic Reviews

Meta-analyses are often used to recover something from poorly designed studies, studies of insignificant statistical power, studies

that give erratic results and those resulting in apparent contradictions.

If a medical treatment has an effect so recondite and obscure as to require meta-analysis to establish it, I would not be happy to have it

used on me. It would seem better to improve treatment, and the theory underlying the treatment".

HJ Eysenck 1995Systematic Reviews

Chalmers I, Altman DG BMJ Publishing Group, London. p73

Limitations of systematic reviews

• Combines studies of varying quality, from the excellent to the appalling

• May be inconsistent with high quality RCTs• Important distinctions between primary studies

may be lost (inclusion/exclusion criteria or intervention)

• Dissimilar conclusions depending the "review question"

• Over-generalisation may make it difficult for practitioners to apply the results

Bias in systematic reviews

• Publication bias– Negative studies tend not to get published– Data from one study can be available in

multiple formats• Selection bias

– inclusion criteria • Language bias

– predominantly English language studies included

When systematic reviews disagree with each other

• Aim

• Methods

• Judgement of quality of studies

• Summing up evidence

Systematic reviews include an element of judgement, whatever method is used

Quality Of Reporting Of Meta-analyses (QOROM)

• Encourages authors to provide readers with information regarding – searches– selection– validity assessment– data abstraction– study characteristics– quantitative data synthesis– trial flow

Qualitative research & SRs

• Cochrane Qualitative Research Methods Group – Aims to provide guidance on methodological

standards – Aims to publish a protocol for conducting SR

of qualitative evidence – There is a place for qualitative research in

reviews of evidence, but how can it be incorporated?

Qualitative Systematic Reviews

• Searching the qualitative literature

• Appraising the qualitative literature

• Synthesizing the qualitative evidence

• Linking with existing quantitative evidence

Appraising qualitative research evidence

• Should we?

• How is it done?

• Criteria – or is this imposing a positivist framework?

• Ultimately a means of determining the quality of qualitative research is needed

Quality criteria

• Many criteria but little agreement

• Some criteria enforce values of the positivist paradigm (Kappa statistics in coding) eg. Engel & Kuzel, 1992; Strauss & Corbin, 1998)

• Is this meaningful? (Morse, 1994; Yardley, 2000)

• What grounds are the criteria based?

Quality

• Philosophical underpinnings impact on criteria (some are incompatible)

• Procedural aspects should be considered

Good quality

• Clear reporting of methods– Sampling– Data collection– Analysis

• Transparency– Adequate presentation of the data

Good quality

• Analysis – Grounded in the data

• Trustworthiness– Integration of research question, methods,

data, results, and conclusions

Further work to develop SRs of Qualitative studies

• Need to consider diverse study designs

• Theoretical perspectives are hugely influential

• Identify fatal flaws

Cochrane or cock-eyed? how should we conduct systematic reviews of qualitative research?

Andrew Booth Senior Lecturer in Evidence Based Healthcare Information, School of Health and Paper presented at the Qualitative Evidence-based Practice Conference, Taking a

Critical Stance. Coventry University, May 14-16 2001

AbstractThe quantitative versus qualitative debate has taken significant steps towards reconciliation within the wider field of evidence based practice. Nevertheless, far more insidious discrimination remains. Systematic review methodology exhibits all the characteristics of "institutionalised quantitativism" in that criteria for a "good" review are almost entirely determined by the quantitative methods promoted and perpetuated by the Cochrane Collaboration. Nobody who understands qualitative research would insist that its primary studies demonstrate alien concepts such as "sample size" or "statistical power". Yet comparably fundamental absurdities persist with regard to qualitative syntheses. Why should systematic reviewers of qualitative research pursue a "gold standard" comprehensive literature search when concepts such as "data saturation" have an established pedigree? Why should they apologise for an absence of meta-analysis when little-known techniques such as meta-ethnography could be included in a reviewer's toolbox? Why shouldn't they apply systematic, explicit and reproducible principles of thematic or concept analysis to create syntheses that advance our understanding of qualitative issues and highlight research gaps? The author draws on experience of a dozen systematic reviews, a third qualitative, to suggest how systematic reviews of qualitative research might acquire a methodology that is more sympathetic to the paradigm within which they are conducted.

Synthesised