Introduction to Systematic Reviews and Meta-Analyses

Craig A Umscheid, MD, MSCE, FACP

Assistant Professor of Medicine and Epidemiology

Director, Penn Center for Evidence-based Practice

Senior Associate Director, ECRI-Penn AHRQ EPC

TEACH Level II Workshop 4

NYAM

August 8th, 2013

PENN CENTER FOR EVIDENCE-BASED PRACTICE

2

Outline

The CaseDefinitionsSteps to Consider when Performing or

Appraising a SR/MAReporting ToolConclusions

3

The Case

4

Setting the Stage: Images of the H1N1 Influenza Epidemic

5

2009 H1N1 Cases Seen in One Mexican Hospital

JAMA. 2009;302(17):1880-1887

6

The Case

You’re the CMO of a moderate sized hospital in Upstate NY, 100s of miles from a quarternary care hospital, and you hear about the below patient from the Director of your MICU:

38 yo physician admitted to your hospital with fever, cough, SOB, and muscle aches worsening over three days

PMH of high cholesterol and seasonal allergies, on a statin for cholesterol SH of 1 drink per day, no tobacco or illicits, 2 kids (3y and 1y), wife, and 2

cats VS: BP 110/70 HR 115 RR 28 O2 sat 90% Multilobar pneumonia on CXR, lab tests diagnostic for H1N1 Admitted to the ICU for hypoxemia Despite intubation and the use of salvage therapies, hypoxemia persists

and patient deteriorates Within first week of his ICU stay, patient codes and despite heroic efforts

by his team he is pronounced dead

7

Demographics of H1N1 Patients in Critical Care

JAMA. 2009;302(17):1880-1887

8

Demographics of Hospitalized Pts with H1N1

Demographics of Hospitalized Patients with H1N1

Study Age (mean)

Co-Morbidities (mean N) Time Course (median days)

Symptoms to hospital

Hospital to ICU

Hospital to Death Mortality

JAMA (Mexico) 44 years 2 6 days 1 day 10 days 41% (60d)

JAMA (Canada) 32 years 2 4 days 1 day 14 days 17% (90d)

JAMA. 2009;302(17):1880-1887

JAMA. 2009;302(17):1872-1879

9

Extra Corporeal Membrane Oxygenation (ECMO)

10

The Case (continued)

Should our moderate-sized hospital invest in an ECMO program?

11

Should our moderate-sized hospital invest in an ECMO program?

A. Yes

B. No

C. Don’t know, need more information

12

Definitions

13

“Narrative review”

Traditional review by content expertMost common method of summarizing a fieldLimitations:

• Lack systematic methods to identify, appraise, and synthesize information

• Potential for selective inclusion and exclusion of studies to support a position

14

“Original research study”

Well designed, well conducted studies can provide unbiased information

Limitations:• False negative results • False positive results • Weak external validity

15

“Systematic reviews” vs. “Meta-analyses”

Systematic review • Literature review

prepared using a systematic approach to minimize bias and random error in identification of information

• May or may not include a meta-analysis

Meta-analysis • Statistical pooling of the

results of individual studies

• Aims to produce a single estimate of treatment effect

16

Systematic Review/Meta-analysis

Address limitations of narrative reviews:• Uses prespecified, explicit methods to:

– reduce bias– allow for replication– reveal areas of uncertainty or in need of further research

Address limitations of original research studies:• Pools data across studies to:

– give more precise estimates of effect– reduce false positive or false negative results– improve generalizability – explore between-study differences in results

BMJ. Volume 315. November 22, 1997.

17

Meta-analysis Medline publications

0

5000

10000

15000

20000

25000

Year

No.

Pub

licat

ions

18

Steps to Consider when Performing or Appraising a SR/MA

19

Seven Key Steps for SR/MA

1. Define the question2. Establish eligibility criteria3. Search literature4. Extract data5. Evaluate individual study quality6. Synthesize data qualitatively and/or

quantitatively (MA)7. Explore heterogeneity

20

1. Define the question Questions most commonly address therapeutic

interventions, harms, and diagnostic tests

Type of question can point to type of studies to seek

Type of question Study Design

Therapy Randomized controlled trial

Harm RCT, cohort study, case-control study

Diagnostic test Cross-sectional

21

1. Define the question (cont)

Population• age range, gender, race, diagnosis, severity class

Intervention or exposure • specific drug or device, doses, frequencies, IV, PO,

inpatient, outpatient, management strategiesComparison group

• placebo, usual care, active comparatorOutcomes

• process measures, clinical outcomes, cost

22

PICO for ECMO SR

Patients• Adults• Acute respiratory failure from H1N1 pneumonia• Any critical care unit in the world

Intervention• ECMO

Comparator• Standard Care

Outcomes• Mortality

23

Seven Key Steps for SR/MA

1. Define the question2. Establish eligibility criteria3. Search literature4. Extract data5. Evaluate individual study quality6. Synthesize data qualitatively and/or

quantitatively (MA)7. Explore heterogeneity

24

2. Establish eligibility criteria

Study designs • RCT, cohort, case-control, cross-sectional

Publication status• Peer-reviewed publications only• Also include abstracts?

Language• English, other languages?

Other• Publication year• Study size• Minimum follow-up

25

A Note on Publication Bias

Bias can be introduced in a SR by not including relevant studies:• Unpublished (Gray Literature)• Published in non-English language• Published only in meeting proceedings or as abstracts• Published but not identified

26

Cumulative Publication Rate for RCTs

Scherer. CDSR. 2007.

27

Submission and Publication Time

Ioannidis. JAMA. 1998.

28

Funnel Plots: Assessing Publication Bias

No Publication Bias Publication Bias

29

Funnel Plot of Effects of Homeopathy

Linde, Lancet, 1997

30

Funnel Plot following Trim and Fill

31

Eligibility criteria for ECMO SR

Only included published studiesStudies were retrieved if:

1. reported on the use of ECMO in patients with influenza OR

2. any controlled trial reporting comparisons between those managed with and without ECMO

Studies had to report mortality ratesMinimum of ten patients in each groupNo language restrictions

32

Seven Key Steps for SR/MA

1. Define the question2. Establish eligibility criteria3. Search literature4. Extract data5. Evaluate individual study quality6. Synthesize data qualitatively and/or

quantitatively (MA)7. Explore heterogeneity

33

3. Search literature

Use librarians to help devise search strategy, and filters to limit your search

Identify information resources to search • Electronic literature databases• Reference lists of key papers and review articles • Hand searches of high value journals• Discussion with experts in the field• Contact pharmaceutical or device manufacturers• Trial registries

Use sampling of resources to increase yield

Screen titles, abstracts and full text• Single versus two independent reviewers?• Automatation?

– Abstrckr. Available at: http://tuftscaes.org/citation_screening/

34

Search Filters

Umscheid. CID. 2013.

35

Electronic Literature Databases

Medline (Medical Literature Analysis and Retrieval System Online) • Premier database of medical research• PubMed interface is free• Ovid interface supported thru institutional accounts, and allows for more

complex searching

Embase (Excerpta Medica Database) • Large European database similar in scope and content to Medline• Includes many conference proceedings• Up to 70% citations in Embase not in Medline

Cochrane Controlled Trials Registry• Fastest, most reliable method for determining if a controlled trial has been

published on any topic

36

Search Strategy for ECMO SR

37

Seven Key Steps for SR/MA

1. Define the question2. Establish eligibility criteria3. Search literature4. Extract data5. Evaluate individual study quality6. Synthesize data qualitatively and/or

quantitatively (MA)7. Explore heterogeneity

38

4. Extract Data

Set priorities for data to extract• Anticipate structure and content of final evidence tables• Resist temptation to extract everything

Establish quality control• Dual extraction vs. solo extraction with random checks

Consider free electronic resources to help• Systematic Review Data Repository. Agency for Healthcare Research

and Quality. 2013. Available at: http://srdr.ahrq.gov/.

39

Data extraction for ECMO SR

Pulmonary

40

Seven Key Steps for SR/MA

1. Define the question2. Establish eligibility criteria3. Search literature4. Extract data5. Evaluate individual study quality6. Synthesize data qualitatively and/or

quantitatively (MA)7. Explore heterogeneity

41

5. Evaluate individual study quality

A study’s quality is a measure of its internal validity

Biased studies lead to erroneous findings in systematic reviews

Decide on how to evaluate quality• Use established scales (e.g. Jadad scale for RCTs)• Select components most likely to distinguish studies with higher risks of bias

from those with lower risks

Components versus scales• Scales have advantage of providing single summary measure of quality • Multiple scales exist, few are validated, and there are differences in items

included and weighting of items across scales• Component analysis has advantage of assessing specific aspects of quality

42

Quality scales

Moher et al. Control Clin Trials. 1995.

43

44

Evaluating quality of ECMO RCTs

45

Study quality (continued)

Decide how quality evaluation will be used• To determine eligibility of study for SR• To weight studies• For subgroup analyses

Subgroup analyses based on quality scales or criteria can help determine effect of bias in studies

46

Examining Effect of LWMH on Risk of DVT Stratified by Study Quality

Nurmohamed. Lancet. 1992.

47

Quality of RCTs in ECMO SR

48

Seven Key Steps for SR/MA

1. Define the question2. Establish eligibility criteria3. Search literature4. Extract data5. Evaluate individual study quality6. Synthesize data qualitatively and/or

quantitatively (MA)7. Explore heterogeneity

49

6. Synthesize data Qualitative synthesis using evidence

tables and written evidence summaries

Quantitative synthesis using meta-analysis• RevMan. Cochrane. 2012. Available at:

http://ims.cochrane.org/revman. • OpenMeta[Analyst]. 2012. Available at:

http://www.cebm.brown.edu/open_meta.

50

Evidence table for ECMO SR

51

Meta-analysis Estimate a summary measure and its

variance for each study Weight each study according to its sample

size - Studies with more “events” get greater weight

Statistically combine or pool results from each study to obtain a weighted average

- Not just a simple average Assess for statistical heterogeneity

52

Forest plot of effect of ECMO on mortality in ARF

2009

53

Assessing heterogeneity

Heterogeneity• Are differences in study-specific estimates statistically significant?

Assessment• Visual examination of Forest Plot• Statistical tests

54

Efficacy of BCG Vaccination

Colditz, JAMA, 1994

55

Statistical tests of heterogeneity Assess whether study-specific estimates are statistically different

from one another

Q test (i.e. chi squared test)• Most common test used• Power is generally low with small number of studies • Excessive power with large number of studies

I squared• Percentage of variation across studies that is due to heterogeneity

between studies and not chance• 0 to 100%

– >50% is considered moderate to high

Importantly, heterogeneity can exist even in absence of statistical significance

BMJ VOLUME 327 6 SEPTEMBER 2003 p557-60

56

Forest plot of effect of ECMO on mortality in ARF

2009

57

Fixed effects vs. Random effects Analyses

Fixed Effects• Assumes there is one underlying effect of intervention which all

individual studies are measuring• Assumes any differences between individual study results is a

consequence of sampling variation (i.e. chance)

Random Effects• Assumes any differences between individual study results is a

consequence of chance as well as real differences in the effects of the intervention across different studies

• Offers a more conservative (less precise) pooled estimate• If there is significant heterogeneity, should analyze using random

effects

58

Forest plot of effect of ECMO on mortality in ARF

2009

59

Seven Key Steps for SR/MA

1. Define the question2. Establish eligibility criteria3. Search literature4. Extract data5. Evaluate individual study quality6. Synthesize data qualitatively and/or

quantitatively (MA)7. Explore heterogeneity

60

7. Explore heterogeneity between studies

Differences in study design, populations, intervention, comparator, outcome definitions and conduct of trials can lead to differing results between studies

Examination of differences can help one understand the conditions most likely to yield positive or negative effects from an intervention

61

Meta-analysis of AZT on Mortality

Search identified 8 studies

7 studies: no effect1 study: large effect

Why do you think 1 study was not pooled with the others?

62

Forest plot of effect of ECMO on mortality in ARF

2009

63

Study heterogeneity

If studies are heterogenous:• “Don’t ignore, instead explore!”

Exploration of heterogeneity is a critical objective of meta-analysis

64

Seven Key Steps for SR/MA

1. Define the question2. Establish eligibility criteria3. Search literature4. Extract data5. Evaluate individual study quality6. Synthesize data qualitatively and/or

quantitatively (MA)7. Explore heterogeneity

65

Reporting Tool for a SR and MA

66

67

68

69

Back to Our Case

Should our moderate-sized hospital invest in an ECMO program?

70

Should our moderate-sized hospital invest in an ECMO program?

A. Yes

B. No

C. Don’t know, need more information

71

Conclusions

SR/MA is scientifically rigorous process of identifying the available evidence on a particular topic

SR are both qualitative and quantitative (MA)

MA involves statistical pooling of available data that results in weighted average

Exploration of heterogeneity in MA is critical

Potential limitations of SR/MA include poor quality of identified studies, publication bias, and heterogeneity

72

References

General Systematic Review Resources Umscheid CA. A primer on performing systematic reviews and meta-analyses. Clinical Infectious Diseases. 2013. Egger M, Davey Smith G, Altman D (eds). Systematic Reviews in Health Care. Meta-analysis in Context. London:

BMJ Books, 2001.

Resources for Systematic Reviews Examining Harm Chou R, Helfand M. Challenges in systematic reviews that assess treatment harms. Annals of Internal Medicine.

2005: 142(12): 1090-1099. Loke YK, Price D, Herxheimer A. Systematic reviews of adverse effects: framework for a structured approach. BMC

Medical Research Methodology. 2007. 7:32. Hammad TA, Pinheiro SP, Neyarapally GA. Secondary use of RCTs to evaluate drug safety: a review of

methodologic considerations. Clinical Trials. 2011.

Resource for Systematic Reviews Examining Diagnostic Tests Smetana GW, Umscheid CA, Chang S, Matchar DB. Methods guide for authors of systematic reviews of medical

tests: a collaboration between the Agency for Healthcare Research and Quality and the Journal of General Internal Medicine. J Gen Intern Med. 2012;27: Suppl 1:S1-3.

73

http://www.uphs.upenn.edu/cep/

My email: craig.umscheid@uphs.upenn.edu