Summarising what we already know – the pivotal role of systematic reviews

Malcolm Macleod

Is there an agreed and accepted definition of systematic reviews?

• Characteristics– A pre-defined question– A pre-defined search strategy– Pre-defined inclusion and exclusion criteria

• May include a meta-analysis, but doesn’t have to

[Are systematic reviews only about the application

of a methodology (bias-reducing principles), or do systematic reviews require a clearly defined topic and

specific question, as well as rigorous inclusion and

exclusion criteria?]

Why bother – what does it add?

1. Less biased research summary– “Positive” findings are more likely in

• Papers compared with Abstracts• High impact journals compared with low impact

journals• English Language verses non-English language

reports• Published versus unpublished findings

The File Drawer problem

0worse better

Characteristics of narrative reviews

• Research summaries based on what comes through your inbox in eTOC alerts do not adequately describe the whole literature

• Data supporting the opinion of the reviewer are more likely to be cited

Why bother – what does it add?

2. Has the work been done before?– Replicating the work of others is an

important part of science– Unknowing replication is a waste of time

(and your money)– Some things may already be known with

sufficient certainty that further experiments are not required (but e.g. positive control)

When and where are systematic reviews most

likely to be carried out and published from now on?

Are they common?

0

10

20

30

40

50

60

70

80

90

1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008

Pubmed search “systematic review” limited to animals conducted 22/1/09

When?

• In drug selection for clinical trial

• In decisions to proceed to clinical trial

• In preparation of grant and ethics applications

• In the planning of research

• In the refinement of the methodologies used in animal experiments

Where, and by whom?

• By people like me interested in developing new treatments

• By basic scientists summarising existing knowledge and planning new experiments

• By people studying the in-use characteristics of animal models of their disease

Publication

• Systematic reviews of clinical trials are widely published – sometimes in specialist journals

• Rigorous systematic reviews are cited about twice as frequently as narrative reviews

• The publishers of the Cochrane Library are exploring the possibility of publishing preclinical reviews

Do we have some examples of systematic reviews in basic

animal research?

• Different purposes …– To survey an area of research– To summarise the effectiveness of a specific

intervention– To examine the impact of potential sources of

bias

To survey an area of research

To summarise the effectiveness of a specific intervention

Hypothermia101 publications277 experiments

3353 animals

Bett

er

Wors

e

-100

0

100

200

To examine the impact of potential sources of bias

NXY 0599 publications29 experiments

408 animalsImproved outcome by

44% (35-53%)

Should the approach of systematic reviews be applied to all areas of

preclinical research?

• It probably could be, but methodologies may need some adaptation according to the specifics of the field

• Horizontal extension– from stroke to related diseases– Parkinsons, MS, Alzheimer’s, Neuro-oncology, Depression

• Vertical extension– Transgenic studies– Pathophysiological studies– In vitro studies

• Its wider application is likely to lead to substantial insights into the characteristics of the models used

Does the application of systematic reviews to

animal research require additional funding, and if

so where should this come from?  

• Yes – but not much – Edinburgh group ~ £6k per publication

• This funding should come from those wishing to improve the usefulness of animal models of human disease

Are many reviews currently being published which are not systematic -- and if so

when, where and why?

Who is responsible for ensuring that reviews

are systematic, and how can we encourage this to

happen?

What questions remain about the validity of

research using animals as models of human disease,

and whether animal research has predictive value for human health?