Ispor workshop 08 april2016
-
Upload
cadth-acmts -
Category
Healthcare
-
view
67 -
download
0
Transcript of Ispor workshop 08 april2016
How can policy-makers and clinicians trust the results of a network meta-analysis? A workshop on how to apply the International Society For Pharmacoeconomics and Outcomes Research (ISPOR) tool
Prepared for : 2016 CADTH Symposium
April 10, 2016
Knowledge Translation Program Li Ka Shing Knowledge InstituteSt. Michael's Hospital Toronto, Canada
Areti Angeliki Veroniki, MSc, PhDSharon E. Straus, MD, FRCPC, MSc
Andrea C. Tricco, MSc, PhD (contributor)
Knowledge Translation, Li Ka Shing Knowledge Institute, St. Michael's Hospital, Toronto, Canada
We have no actual or potential conflict of interest in relation to this presentation
Knowledge Translation, Li Ka Shing Knowledge Institute, St. Michael's Hospital, Toronto, Canada
of the presentation
• To gain knowledge about what a network meta-analysis (NMA) is • What is the definition of NMA and how is it related to
pairwise MA?
• To gain knowledge and skills in assessing the validity of an NMA • How do we know whether the NMA is credible?
• To work in small groups to establish the credibility of a NMA • Do we trust the results of this NMA?• Is there anything missing?• Could the authors have done things differently?
3
Knowledge Translation, Li Ka Shing Knowledge Institute, St. Michael's Hospital, Toronto, Canada
Network Meta-analysis (NMA)
NMA is an extension
of pairwise meta-
analysis that
simultaneously
compares multiple
treatments for a
medical condition
from two or more
studies that have one
treatment in common
Knowledge Translation, Li Ka Shing Knowledge Institute, St. Michael's Hospital, Toronto, Canada
14 serotonin (5-HT3) receptor antagonists combinations for vomiting for patients undergoing surgery
? Placebo
Ondansetron
Granisetron
DolasetronTropisetron
Ondansetron+Dexamethasone
Palonosetron
Ramosetron
Ondansetron+DroperidolIV
Ondan+Metoclopr IV
Granisetron+Dexamethasone
Palonosetron+DexamethasoneDolasetron+Dexamethasone
Dolasetron+DroperidolIV
Granisetron+DroperidolIV
5
238 RCTs and 33 meta-analyses Tricco et al BMC Medicine 2015
Knowledge Translation, Li Ka Shing Knowledge Institute, St. Michael's Hospital, Toronto, Canada
Terminology• Network meta-analysis (or multiple treatment meta-analysis or
mixed treatment comparison)o Combines direct and indirect data across a network of studies to
infer the relative effectiveness/safety of ≥3 interventions in a single model
• Indirect comparisono Allows the estimation of the relative effectiveness/safety of ≥2
treatments in the absence of head-to-head evidence
• Mixed comparisono Combines both direct and indirect evidence for a specific
comparison to obtain a weighted average of the estimates for a single treatment comparison at the time
Salanti Res Syn Meth 2012
Knowledge Translation, Li Ka Shing Knowledge Institute, St. Michael's Hospital, Toronto, Canada
7
Network Meta-analysis (NMA)
Nikolakopoulou et al PLoS One 2013 Number of publications
1997
2000
2002
2003
2004
2005
2006
2007
2008
2009
2010
2011
2012
0 10 20 30 40 50
The number of published systematic reviews that employ NMA are increasing over time.
Given the relevance of NMA to inform healthcare decision-making, it is of great interest to improve
understanding of these studies by decision-makers and policy-makers.
Knowledge Translation, Li Ka Shing Knowledge Institute, St. Michael's Hospital, Toronto, CanadaExample – NMA published in BMJ
OpenPlease read the following paper
Tricco et al BMJ Open 2015
Knowledge Translation, Li Ka Shing Knowledge Institute, St. Michael's Hospital, Toronto, Canada
Jansen et al 2014
International Society for Pharmacoeconomics and Outcomes
Research (ISPOR) checklist
Knowledge Translation, Li Ka Shing Knowledge Institute, St. Michael's Hospital, Toronto, Canada
Questionnaire Items – 2 main domains
Relevance (4 questions)
Credibility (22 questions)
Analysis (7 questions)
Reporting quality & transparency (6 questions)
Interpretation (1 question)
Conflict of interest (2 questions)
Evidence Base (6 questions)
Knowledge Translation, Li Ka Shing Knowledge Institute, St. Michael's Hospital, Toronto, Canada
Scoring
• Credibility and Relevance are scored separately
• Questions are grouped by domain and each question is answered as ‘yes’/’no’/’unclear’
• Each domain can be rated as ‘strong’/‘weak’/’neutral’
Knowledge Translation, Li Ka Shing Knowledge Institute, St. Michael's Hospital, Toronto, Canada
Evidence base1. Did the researchers attempt to identify and include all
relevant RCTs?• Did the search strategy include terms for RCTs of all interventions
of interest?• Were multiple databases searched (e.g., MEDLINE, EMBASE, and
Cochrane Central Registry of Trials)?• Do the inclusion criteria include all RCTs of interest (if identified
by the literature search)?– Language limitations, search for unpublished material
A ‘yes’ to the above implies an adequate attempt to include all available RCTs
Knowledge Translation, Li Ka Shing Knowledge Institute, St. Michael's Hospital, Toronto, Canada
Systematic Review Conduct– The same considerations as in a traditional systematic review
and meta-analysis still apply!• A well-conducted systematic review follows the Cochrane Collaboration
methodology
– Importance of clinical input on the systematic review conduct • Need to ensure the team includes a range of expertise in systematic
review methods, clinical, and statistical domains
– Arguably more important to have a well-conducted underlying systematic review for NMA
• NMA used to base policy decisions on the population level = affect many more people
Knowledge Translation, Li Ka Shing Knowledge Institute, St. Michael's Hospital, Toronto, Canada
A Systematic Review usually has….
o Comprehensive (≥2 databases) and PRESS’ed literature search
o Pre-defined inclusion and exclusion criteria (i.e., study eligibility criteria)
o Risk of bias appraisal (Cochrane tool for randomized clinical trials)
o Pre-defined data abstraction form
o Synthesis based on the totality of evidence and reported using PRISMA
o Discussion, providing limitations of included studies and review process
o Protocol using PRISMA-P, PROSPERO registry, published in an open access journal (e.g., Sys Rev journal, BMJ Open)
o Each step conducted by 2 reviewers, independently
Cochrane Handbook (editors: Higgins and Green) 2011, Shamseer BMJ 2015, Sampson JCE 2009, Moher BMJ 2009
Knowledge Translation, Li Ka Shing Knowledge Institute, St. Michael's Hospital, Toronto, Canada
Evidence base1. TASK: Did the researchers attempt to identify and include
all relevant RCTs?
Knowledge Translation, Li Ka Shing Knowledge Institute, St. Michael's Hospital, Toronto, Canada
Evidence base2. Do the trials for the interventions of interest form one
connected network of RCTs?• Network geometry and connectedness
No_treat
ZIDO,AZT,ZDV
ZDV+3TC
Any_ARTZDV+3TC+ABC
ZDV+3TC+NEVI
HAART
Any_treat
NEVI
EFAV
LAMIV,3TCTENOF
d4T+3TC
ZDV+ddI+NEVI
No_treat
ZIDO,AZT,ZDV
ZDV+3TC
Any_ART
ZDV+3TC+ABC
ZDV+3TC+NEVI
HAART
TENOF d4T+3TC
ZDV+ddI+NEVI
Knowledge Translation, Li Ka Shing Knowledge Institute, St. Michael's Hospital, Toronto, Canada
Evidence base2. TASK: Do the trials for the interventions of interest form
one connected network of RCTs?
Knowledge Translation, Li Ka Shing Knowledge Institute, St. Michael's Hospital, Toronto, Canada
Evidence base3. Is it apparent that poor quality studies were included,
thereby leading to bias?• Cochrane ROB (Sensitivity analysis by components of ROB)
7
6
5
4
3
2
1
9%
39%
74%
98%
98%
16%
37%
39%
55%
14%
2%
1%
84%
63%
52%
5%
13%
1%
0%
0%
Low Unclear High
1 2 3 4 5 6 7Random Sequence
generationAllocation
concealmentBlinding of participants
and personnelBlinding of outcome
assessmentIncomplete
outcome dataSelective reporting
Other bias
Knowledge Translation, Li Ka Shing Knowledge Institute, St. Michael's Hospital, Toronto, Canada
Evidence base3. Is it apparent that poor quality studies were included,
thereby leading to bias?• Cochrane ROB (Sensitivity analysis by components of ROB)
Placebo
BUDE
FORMINDAC
SAML
ACLI
GLYC
TIOTFORM/BUDE VILA/FLUT
SALM/FLUT
INDA/GLYC
Randomization
Placebo
BUDE
FORMINDAC
SAML
ACLI
GLYC
TIOTFORM/BUDE VILA/FLUT
SALM/FLUT
INDA/GLYC
Allocation Concealment
Knowledge Translation, Li Ka Shing Knowledge Institute, St. Michael's Hospital, Toronto, Canada
Evidence base3. TASK: Is it apparent that poor quality studies were included,
thereby leading to bias?
Knowledge Translation, Li Ka Shing Knowledge Institute, St. Michael's Hospital, Toronto, Canada
Evidence base4. Is it likely that bias was induced by selective reporting of
outcomes in the studies?• Use the Cochrane ROB tool to help ascertain this
0.5
Sta
ndar
d er
ror o
f log
odd
s ra
tio
-2 -1 0 1 2Log-odds ratio centred at comparison-specific pooled effect
Comparison adjusted funnel plot
Tricco et al BMJ 2014; Chaimani et al Plos One 2013
Asymmetry
Knowledge Translation, Li Ka Shing Knowledge Institute, St. Michael's Hospital, Toronto, Canada
Evidence base4. TASK: Is it likely that bias was induced by selective reporting
of outcomes in the studies?
Knowledge Translation, Li Ka Shing Knowledge Institute, St. Michael's Hospital, Toronto, Canada
Evidence base5. Are there systematic differences in treatment effect
modifiers across the different treatment comparisons in the network?• Relates to baseline patient or study characteristics that have an impact on
the treatment effects (transitivity assumption)
20 25 30
20 25 3020 25 30
20 25 30
A
B
C
A
B
CInvalid Indirect Comparison × Valid Indirect
Comparison
Age effect modifier
Knowledge Translation, Li Ka Shing Knowledge Institute, St. Michael's Hospital, Toronto, Canada
Evidence base5. Are there systematic differences in treatment effect
modifiers across the different treatment comparisons in the network?• Relates to baseline patient or study characteristics that have an impact on
the treatment effects (transitivity assumption)
A
B
C
C
T
O
P
P
Ondasetron
P-InjectionP-Pill
Plac
ebo Might be an
inappropriate common comparator
Tropisetron
Knowledge Translation, Li Ka Shing Knowledge Institute, St. Michael's Hospital, Toronto, Canada
Evidence base5. Are there systematic differences in treatment effect
modifiers across the different treatment comparisons in the network?
Placebo
Ondan-4mgOndan-8mg
Ondan-16mgOndan-12mg
Ondan-1mgOndan-3mg
Ondan-24mgOndan-30mg
Ondan-32mgOndan-48mg
Ondan-2mgOndan-10mgOndan-6mg
Ondan-5mgOndan-0.2mgOndan-9mg
Grani-0.1mgGrani-0.2mg
Grani-0.3mgGrani-1mg
Grani-2mgGrani-3mg
Grani-2.5mgGrani-2.8mg
Grani-0.6mgGrani-1.2mg
Grani-0.4mgGrani-1.1mgGrani-0.7mgGrani-2.2mgGrani-0.8mg
Dola-12.5mgDola-25mgDola-37.5mg
Dola-7.0mgDola-54.0mg
Tropi-2mgTropi-5mgTropi-0.5mg
Tropi-0.1mgTropi-1mg
Tropi-1.5mgTropi-7.3mg
Tropi-4.3mgPalono-0.025mgPalono-0.05mgPalono-0.075mg
Palono-0.008mgPalono-0.021mg
Palono-0.074mgPalono-0.219mg
Palono-2.130mgPalono-0.25mgRamo-0.3mgRamo-0.6mgRamo-0.1mgRamo-0.9mgRamo-0.2mgPlacebo
OndasetronGranisetro
n
Dolasetron
Tropisetron
Palonosetron Ramosetron
Lumping or splitting nodes?
Knowledge Translation, Li Ka Shing Knowledge Institute, St. Michael's Hospital, Toronto, Canada
Evidence base5. Are there systematic differences in treatment effect
modifiers across the different treatment comparisons in the network?• ‘Missing’ arms are missing at random: Observed and unobserved
estimates do not differ beyond what can be explained by heterogeneity
Study Observed
AC
AB
Study If arm were included…
Observed and Unobserved
AC B
AB C
Lu and Ades 2006B
C
A
C
B
B might have been used in trials in the 1990s , whereas C in 2000s
Knowledge Translation, Li Ka Shing Knowledge Institute, St. Michael's Hospital, Toronto, Canada
Evidence base5. Are there systematic differences in treatment effect
modifiers across the different treatment comparisons in the network?• Interpretation of transitivity:
1. Treatment A is similar when it appears in AB and AC trials
2. The two sets of trials AB and AC do not differ with respect to the distribution of effect modifiers.
3. Participants included in the network could in principle be randomized to any of the 3 treatments A, B, C
4. ‘Missing’ treatment in each trial is missing at random
5. There are no differences between observed and unobserved relative effects of AB and AC beyond what can be explained by heterogeneity
Salanti Res Synth Methods 2012
B
C
A
Knowledge Translation, Li Ka Shing Knowledge Institute, St. Michael's Hospital, Toronto, Canada
Evidence base5. TASK: Are there systematic differences in treatment effect
modifiers across the different treatment comparisons in the network?
Hint: The answer to question 5 is a “yes” if there are substantial (or systematic) differences in effect modifiers, which can be judged by comparing study-specific inclusion and exclusion criteria, baseline patient characteristics, and study characteristics that are expected to be effect modifiers.
Knowledge Translation, Li Ka Shing Knowledge Institute, St. Michael's Hospital, Toronto, Canada
Evidence base6. If yes, were these imbalances in effect modifiers across the
different treatment comparisons identified before comparing individual study results?
It is recommended:a) Before undertaking a NMA, generate a list of potential treatment
effect modifiersb) Next, compare the suggested treatment effect modifiers across
studies to identify any imbalances between the different types of direct comparisons in the network.
Knowledge Translation, Li Ka Shing Knowledge Institute, St. Michael's Hospital, Toronto, Canada
Evidence base6. If yes, were these imbalances in effect modifiers across the
different treatment comparisons identified before comparing individual study results?
Orange: no pregnant women
NPH[od/bid]
NPH[od]
D[od/bid]
D[qid]
D [od]G[bid] G[od]
Placebo
O
G
DTO+D
P
R
O+D_IV
O+M_IV
G+DexP+Dex
D+Dex
D+D_IV
G+D_IV
1: children (orange), 2:adults (purple), 3:all (red)
Knowledge Translation, Li Ka Shing Knowledge Institute, St. Michael's Hospital, Toronto, Canada
Evidence base6. If yes, were these imbalances in effect modifiers across the
different treatment comparisons identified before comparing individual study results?
Treatment Comparison RoB - Allocation concealment RoB - Incomplete outcome data Age AD severity Comorbidities
DONE vs PLAC High High >75 Moderate-Severe 0RIVA vs PLAC Low Low <75 Mild-Moderate 0GALA vs DONE Unclear Unclear <75 Mild-Moderate 0RIVA vs DONE Low Low <75 Mild-Moderate 0
DONE vs PLAC Unclear High <75 Moderate 0GALA vs PLAC Low High <75 Moderate 0RIVA_O vs PLAC Low High <75 Mild-Moderate 0RIVA_P vs PLAC Low High <75 Moderate 0MEMA vs PLAC High High <75 Moderate-Severe 0GALA vs DONE Unclear High <75 Mild-Moderate 0RIVA_O vs DONE Unclear High <75 Mild-Moderate 0RIVA_O vs GALA Unclear Unclear <75 Mild-Moderate 0RIVA_P vs RIVA_O Unclear High >75 Mild-Severe 0
Bradycardia outcome
Diarrhea outcome
Knowledge Translation, Li Ka Shing Knowledge Institute, St. Michael's Hospital, Toronto, Canada
Evidence base6. TASK: If yes, were these imbalances in effect modifiers
across the different treatment comparisons identified before comparing individual study results?
Knowledge Translation, Li Ka Shing Knowledge Institute, St. Michael's Hospital, Toronto, Canada
Validity [or Credibility as per ISPOR]
Knowledge Translation, Li Ka Shing Knowledge Institute, St. Michael's Hospital, Toronto, Canada
Analysis7. Were statistical methods used that preserve within-study
randomization?• No naïve indirect comparisons
A
B
C
Although NMA is based on RCTs, randomization does not hold across the set of trials used for the analysis because patients are not randomized to different trials.
Knowledge Translation, Li Ka Shing Knowledge Institute, St. Michael's Hospital, Toronto, Canada
Analysis7. Were statistical methods used that preserve within-study
randomization?
Treatment comparison Studies
NPH[od/bid] Detemir[od/bid] 6
NPH[od/bid] Glargine[od] 2
NPH[od] Detemir[od] 4
Detemir[od/bid] Glargine[od] 1
Detemir[qid] Glargine[od] 1
Detemir[od] Glargine[od] 1
Glargine[bid] Glargine[od] 1
NPH[od] Glargine[od] -
Tricco et al BMJ 2014 NPH: Neutral Protamine Hagedorn, od: once daily, bid:twice daily
Interventions for type 1 diabetes and severe hypoglycemia: series of pairwise meta-analyses
Knowledge Translation, Li Ka Shing Knowledge Institute, St. Michael's Hospital, Toronto, Canada
Neutral Protamine Hagedorn
[once / twice daily]
Neutral Protamine Hagedorn [once daily]
Detemir [once daily / twice daily]
Detemir [four times daily]
Detemir [once daily]
Glargine [twice daily]
Glargine [once daily]
6
4
2
1
1
1
1
Tricco et al BMJ 2014 36
Knowledge Translation, Li Ka Shing Knowledge Institute, St. Michael's Hospital, Toronto, Canada
How to compare NPH[od] to Glargine[od]?
Neutral Protamine Hagedorn [once daily]
Detemir [once daily] Glargine [once daily]
2
1
?
Tricco et al BMJ 2014 37
Knowledge Translation, Li Ka Shing Knowledge Institute, St. Michael's Hospital, Toronto, Canada
Indirect Comparison
NPH [od]
Glargine [od]
Detemir [od]
Comparison Odds ratio (OR) 95% Confidence Interval (CI)
NPH[od] vs Detemir[od] 1.72 (1.20, 2.50)Glargine[od] vs
Detemir[od] 2.21 (0.63, 7.77)
How to compare NPH[od] to Glargine[od]?Estimate indirect OR and 95% CI
38Bucher et al J Clin Epidemiol 1997
Knowledge Translation, Li Ka Shing Knowledge Institute, St. Michael's Hospital, Toronto, Canada
1. Indirect treatment effect:
2. Variance=((High CI – Low CI)/3.92)2
3. 95% CI for the indirect estimate:
=[0.28, 0.92] =[-0.46, 2.05]
=
Indirect Comparison
39NG:NPH [od] vs Glargine [od] , ND:NPH [od] vs Detemir [od] , GD: Glargine [od] vs Detemir [od]
=0.04 =0.41
==0.45
=
Knowledge Translation, Li Ka Shing Knowledge Institute, St. Michael's Hospital, Toronto, Canada
Unadjusted/Naïve Indirect ComparisonTreatment A Treatment B
Treatment A Treatment C
𝑅 𝑅𝐵𝑣𝑠𝐶=
1333212431
23674
=2.27
Naïve Indirect Comparison
Adjusted Indirect Comparison
𝑅 𝑅𝐵𝑣𝑠𝐶=0.30
Should be avoided!
40
Knowledge Translation, Li Ka Shing Knowledge Institute, St. Michael's Hospital, Toronto, Canada
Indirect OR for NPH[od] vs. Glargine[od]: 0.78 [0.40, 1.52]
Neutral Protamine Hagedorn
[once / twice daily]
Neutral Protamine Hagedorn [once daily]
Detemir [once daily / twice daily]
Detemir [four times daily]
Detemir [once daily]
Glargine [twice daily]
Glargine [once daily]
6
4
2
1
1
1
13
41
Knowledge Translation, Li Ka Shing Knowledge Institute, St. Michael's Hospital, Toronto, Canada
Indirect and mixed effects
B
C
A
B
C
Indirect effectMixed effect
Direct effectOdds ra
tio
0.01 0.11
10
Odds ratio
0.01 0.11
10
Odds ratio0.01 0.1 1 10
42
B
C
Knowledge Translation, Li Ka Shing Knowledge Institute, St. Michael's Hospital, Toronto, Canada
𝑉𝑎𝑟 (𝑀𝑖𝑥𝑒𝑑 𝐿𝑜𝑔𝑂𝑅)=1
1𝑉𝑎𝑟 (𝐿𝑜𝑔𝑂 𝑅𝐷𝑖𝑟𝑒𝑐𝑡 )
+1
𝑉𝑎𝑟 (𝐿𝑜𝑔𝑂 𝑅𝐼𝑛𝑑𝑖𝑟𝑒𝑐𝑡)
𝑀𝑖𝑥𝑒𝑑 𝐿𝑜𝑔𝑂𝑅=
𝐿𝑜𝑔𝑂 𝑅𝐷𝑖𝑟𝑒𝑐𝑡
𝑉𝑎𝑟 (𝐿𝑜𝑔𝑂𝑅𝐷𝑖𝑟𝑒𝑐𝑡 )+
𝐿𝑜𝑔𝑂 𝑅𝐼𝑛𝑑𝑖𝑟𝑒𝑐𝑡
𝑉𝑎𝑟 (𝐿𝑜𝑔𝑂 𝑅𝐼𝑛𝑑𝑖𝑟𝑒𝑐𝑡)1
𝑉𝑎𝑟 (𝐿𝑜𝑔𝑂𝑅𝐷𝑖𝑟𝑒𝑐𝑡 )+ 1
𝑉𝑎𝑟 (𝐿𝑜𝑔𝑂 𝑅𝐼𝑛𝑑𝑖𝑟𝑒𝑐𝑡)
Mixed comparison
Meta-analysis Pooled Effect
Effect estimate and
variance
43Salanti Res Synth Methods 2012
Mixed Comparison
Direct Comparison
Indirect ComparisonLog odds ratio
0
-1.560
1.56
Summarize direct and indirect effect size into a single
mixed effect
Knowledge Translation, Li Ka Shing Knowledge Institute, St. Michael's Hospital, Toronto, Canada
Mixed comparisonNPH [od]
Detemir [od] Glargine [od]
6
4
21
11
13
We gain precision
NPH[od] Vs. Glargine[od] LogOR [Variance]
Direct Comparison 0.10 [0.20]
Indirect Comparison -0.25 [0.45]
Mixed Comparison -0.008 [0.14]
44
Knowledge Translation, Li Ka Shing Knowledge Institute, St. Michael's Hospital, Toronto, Canada
Extend the idea of mixed effect sizes in the entire network
Neutral Protamine Hagedorn
[once / twice daily]
Neutral Protamine Hagedorn [once daily]
Detemir [once daily / twice daily]
Detemir [four times daily]
Detemir [once daily]
Glargine [twice daily]
Glargine [once daily]
6
4
2
1
1
1
13
45
Knowledge Translation, Li Ka Shing Knowledge Institute, St. Michael's Hospital, Toronto, Canada
Extend the idea of mixed effect sizes in the entire network
Placebo
Ondasetron
Granisetron
Dolasetron
TropisetronOnd+De
x
Palonosetron
Ramosetron
Ond+Drop IV
Ond+Metoclop IV
Gran+Dex Palon+De
x
Dolas+Dex
Dolas+Drop IV
Gran+Drop IV
You need to use more sophisticated techniques to
simultaneously analyze evidence coming from the
entire network
Knowledge Translation, Li Ka Shing Knowledge Institute, St. Michael's Hospital, Toronto, Canada
Comprehensive use of all available data
Avoids selective use of indirect evidence
Comparison of interventions which haven’t been directly compared in any experiment
Network meta-analysis
Plac
Ond
Gran
DolasTrop
Ond+DexPalon
Ramos
Ond+Drop IV
Ond+Metoclop IV
Gran+DexPalon+Dex
Dolas+Dex
Dolas+Drop IV
Gran+Drop IV
47
Knowledge Translation, Li Ka Shing Knowledge Institute, St. Michael's Hospital, Toronto, Canada
Comprehensive use of all available data
Avoids selective use of indirect evidence
Comparison of interventions which haven’t been directly compared in any experiment
It can increase precision in the estimated treatment effects
Network meta-analysis
48
Knowledge Translation, Li Ka Shing Knowledge Institute, St. Michael's Hospital, Toronto, Canada
Network meta-analysis
Ondansetron vs Placebo
Granisetron vs Placebo
Dolasetron vs Placebo
Tropisetron vs Placebo
Ondansetron+DEX vs Placebo
Palonosetron vs Placebo
Ramosetron vs Placebo
Ondansetron+DROP vs Placebo
Ondansetron+METO vs Placebo
Granisetron+DEX vs Placebo
Dolasetron+DEX vs Placebo
Dolasetron+DROP vs Placebo
Granisetron+DROP vs Placebo
Treatment Comparison
0.35 (0.32, 0.39)0.36 (0.33, 0.40)0.24 (0.16, 0.34)0.26 (0.21, 0.34)0.42 (0.21, 0.83)0.44 (0.30, 0.63)0.32 (0.22, 0.48)0.32 (0.23, 0.43)0.16 (0.09, 0.27)0.16 (0.12, 0.23)0.53 (0.38, 0.73)0.38 (0.24, 0.60)0.42 (0.26, 0.68)0.28 (0.18, 0.43)0.15 (0.07, 0.31)0.14 (0.08, 0.26)0.16 (0.06, 0.43)0.15 (0.06, 0.42)0.16 (0.08, 0.31)0.15 (0.09, 0.24)0.06 (0.01, 0.30)0.18 (0.06, 0.49)0.16 (0.07, 0.35)0.19 (0.07, 0.52)0.30 (0.05, 1.66)0.31 (0.11, 0.82)
0.35 (0.32, 0.39)0.36 (0.33, 0.40)0.24 (0.16, 0.34)0.26 (0.21, 0.34)0.42 (0.21, 0.83)0.44 (0.30, 0.63)0.32 (0.22, 0.48)0.32 (0.23, 0.43)0.16 (0.09, 0.27)0.16 (0.12, 0.23)0.53 (0.38, 0.73)0.38 (0.24, 0.60)0.42 (0.26, 0.68)0.28 (0.18, 0.43)0.15 (0.07, 0.31)0.14 (0.08, 0.26)0.16 (0.06, 0.43)0.15 (0.06, 0.42)0.16 (0.08, 0.31)0.15 (0.09, 0.24)0.06 (0.01, 0.30)0.18 (0.06, 0.49)0.16 (0.07, 0.35)0.19 (0.07, 0.52)0.30 (0.05, 1.66)0.31 (0.11, 0.82)
Odds Ratio (95% CI)
1.01 1 100
Tricco et al 2015 BMC Medicine
Treatments for 5ht3 surgery data
Network Estimates
Direct Estimates
We gain precision
49
Knowledge Translation, Li Ka Shing Knowledge Institute, St. Michael's Hospital, Toronto, Canada
Comprehensive use of all available data
Avoids selective use of indirect evidence
Comparison of interventions which haven’t been directly compared in any experiment
It can increase precision in the estimated treatment effects
It can rank all competing treatments for the same condition
Network meta-analysis
50
Knowledge Translation, Li Ka Shing Knowledge Institute, St. Michael's Hospital, Toronto, Canada
0.2
.4.6
.81
0.2
.4.6
.81
0.2
.4.6
.81
0.2
.4.6
.81
0.2
.4.6
.81
0.2
.4.6
.81
0.2
.4.6
.81
0.2
.4.6
.81
0.2
.4.6
.81
0.2
.4.6
.81
0.2
.4.6
.81
0.2
.4.6
.81
0.2
.4.6
.81
0.2
.4.6
.81
0.2
.4.6
.81
1 3 5 7 9 11 13 15 1 3 5 7 9 11 13 15 1 3 5 7 9 11 13 15 1 3 5 7 9 11 13 15
1 3 5 7 9 11 13 15 1 3 5 7 9 11 13 15 1 3 5 7 9 11 13 15 1 3 5 7 9 11 13 15
1 3 5 7 9 11 13 15 1 3 5 7 9 11 13 15 1 3 5 7 9 11 13 15 1 3 5 7 9 11 13 15
1 3 5 7 9 11 13 15 1 3 5 7 9 11 13 15 1 3 5 7 9 11 13 15
Dolasetron Dolasetron+Dexamethasone Dolasetron+DroperidolIV Granisetron
Granisetron+Dexamethasone Granisetron+DroperidolIV Ondansetron Ondansetron+Dexamethasone
Ondansetron+DroperidolIV Ondansetron+MetoclopramideIV Palonosetron Palonosetron+Dexamethasone
Placebo Ramosetron Tropisetron
Pro
babi
litie
s of
eac
h ra
nk
Rank
Network meta-analysis
Tricco et al 2015 BMC Medicine
Treatment SUCRA Mean Rank
Ond+Drop IV 85.4 3Gran+Dex 84 3
Ond+Dex 79.5 4Ond+Met IV 78.6 4
Dol+Dex 71.8 5Dola+Drop IV 68.2 6
Gran 54.4 7Ramos 49.2 8Tropis 41.6 9
Gran+Drop IV 44.6 9Ond 30.8 11
Palon 29.1 11 Dolas 20.8 12
Placebo 4.6 14Palon+Dex 7.5 14
51
Knowledge Translation, Li Ka Shing Knowledge Institute, St. Michael's Hospital, Toronto, Canada
Analysis7. TASK: Were statistical methods used that preserve within-
study randomization?
Knowledge Translation, Li Ka Shing Knowledge Institute, St. Michael's Hospital, Toronto, Canada
Analysis8. If both direct and indirect comparisons are available, was
agreement in treatment effects evaluated or discussed?• Relates to consistency
Indirect evidence
Direct evidence
Are the results valid?Network
Meta-analysis Results
B
C
A
B
C
If all three A, B and C are transitive then the loop is consistent
Knowledge Translation, Li Ka Shing Knowledge Institute, St. Michael's Hospital, Toronto, Canada
Network meta-analysis
Nikolakopoulou et al PLoSOne 2013
01020304050Number of publications
Bayesian hierarchical model
Adjusted Indirect ComparisonNot reported
Meta-regression
0 10 20 30 40 50
1997
2000
2002
2003
2004
2005
2006
2007
2008
2009
2010
2011
2012
Full networkStar network
Systematic reviews that employ NMA are undertaken and published with increasing frequency.
BUT! The validity of the results from NMA rests on the assumption of transitivity!
54
Knowledge Translation, Li Ka Shing Knowledge Institute, St. Michael's Hospital, Toronto, Canada
B
C
A
Consistency = transitivity across a loopThe TRANSITIVITY assumption states that:‘the benefit of A over B’ =
‘the benefit of A over C’ +
‘the benefit of C over B’
The CONSISTENCY assumption states that:‘the overall treatment effect in AB studies’ =
‘overall treatment effect in AC studies’ +
‘overall treatment effect in CB studies’
Untestable assumption
Testable assumption
55
When the common comparator is transitive, it allows a valid comparison of the treatments to which it is linked
Knowledge Translation, Li Ka Shing Knowledge Institute, St. Michael's Hospital, Toronto, CanadaAssumption underlying indirect comparison
and NMA (in addition to considering homogeneity)
Assumption for indirect and mixed comparison
Conceptual definition
(Transitivity)
Clinical Methodological
Property of parameters and
data (Consistency)
Statistical
Cipriani et al Ann of Int Medicine 2013 56
Knowledge Translation, Li Ka Shing Knowledge Institute, St. Michael's Hospital, Toronto, Canada
B
C
A
B
C
Assumption of consistency
Direct and indirect
evidence are in agreement
Consistency is a property of a ‘closed loop’ - a path that starts and ends at the same node.
57
Knowledge Translation, Li Ka Shing Knowledge Institute, St. Michael's Hospital, Toronto, Canada
010
2030
40N
umbe
r of p
ublic
atio
ns
1997
2000
2002
2003
2005
2006
2007
2008
2009
2010
2011
2012
Appropriate statistical methods
Inappropriate methodsNone reported
A database of 186 NMAs showed that… In 24% of the networks the authors used inappropriate methods to evaluate consistency
- Comparison of direct with NMA estimates- Comparison of previous meta-analyses with NMA
results In 44% of the networks the authors did not report a method to evaluate consistency
Network Meta-analysis (NMA)
Nikolakopoulou et al PLoS One 2013 58
Knowledge Translation, Li Ka Shing Knowledge Institute, St. Michael's Hospital, Toronto, Canada
Forms of InconsistencyLoop Inconsistency
AC
AB
Direct AB
Indirect ABB
C
A
BC
Lu and Ades JASA 2006
If they statistically differ :Inconsistency!
59
Knowledge Translation, Li Ka Shing Knowledge Institute, St. Michael's Hospital, Toronto, Canada
Forms of InconsistencyDesign Inconsistency
AB
ABDesign AB
Design ABC
If they statistically differ :
B
C
A
Inconsistency!White et al RSM 2012Higgins et al RSM 2012 60
Knowledge Translation, Li Ka Shing Knowledge Institute, St. Michael's Hospital, Toronto, Canada
Approaches for evaluating…LOCAL INCONSISTENCY Loop-Specific (LS) Node-splitting / Separating Indirect and Direct Evidence (SIDE) Separating One Design from the Rest (SODR)
GLOBAL INCONSISTENCY Composite test for inconsistency Lu and Ades (LA) Design by treatment interaction (DBT)
Note: There is also Comparison of model fit and parsimony between consistency and inconsistency models approach
- Requires Bayesian framework – uses the measures of model fit & parsimony (e.g. DIC)
- Does not provide inconsistency estimates- Infers on global inconsistency 61
Knowledge Translation, Li Ka Shing Knowledge Institute, St. Michael's Hospital, Toronto, Canada
Properties of the inconsistency approaches
Loop-Specific
SIDE/Node splitting SODR Composite
test LA DBT
Simple to compute Insensitive to parameterization of multi-arm studies
Indirect estimate derived from the entire network
Does not suffer from multiple testing Power ? ? ?
62 Song et al BMC Med Res Methodol 2012, Veroniki et al BMC Med Res Methodol 2014
Knowledge Translation, Li Ka Shing Knowledge Institute, St. Michael's Hospital, Toronto, Canada
• Lower statistical heterogeneity is associated with greater chance to detect inconsistency but the estimated magnitude of inconsistency is lower
• Low power: ‘Absence of evidence is not evidence of absence’• The methodological and clinical plausibility of the consistency assumption
should be further considered
• The lack of direct evidence (‘open’ loops) makes the statistical evaluation of consistency impossible • But the transitivity assumption is still needed to derive the indirect estimate!
Issues with statistical estimation of consistency
Song et al BMC Med Res Methodol 2012, Veroniki et al IJE 2013, Veroniki et al BMC Med Res Methodol 2014
1
2
3
4
5
6
7
8
910
11
12
Consistent?
Knowledge Translation, Li Ka Shing Knowledge Institute, St. Michael's Hospital, Toronto, Canada
• Lower statistical heterogeneity is associated with greater chance to detect inconsistency but the estimated magnitude of inconsistency is lower.
• Low power: ‘Absence of evidence is not evidence of absence’• The methodological and clinical plausibility of the consistency assumption
should be further considered
• The lack of direct evidence (‘open’ loops) makes the statistical evaluation of consistency impossible • But the transitivity assumption is still needed to derive the indirect estimate!
• Results and inferences on the prevalence of inconsistency are sensitive to the estimation method of heterogeneity
o Magnitude of heterogeneityo Random/Fixed effects to derive treatment effect estimateso Estimation method for heterogeneity (DL, REML, SJ etc)o Same or different heterogeneity across comparisons in the network
Issues with statistical estimation of consistency
Song et al BMC Med Res Methodol 2012, Veroniki et al IJE 2013, Veroniki et al BMC Med Res Methodol 2014
Knowledge Translation, Li Ka Shing Knowledge Institute, St. Michael's Hospital, Toronto, Canada
Inconsistency - Heterogeneity
Direct AB
Indirect AB
AC
BC
Important heterogeneity might decrease the prevalence of inconsistency!
Knowledge Translation, Li Ka Shing Knowledge Institute, St. Michael's Hospital, Toronto, CanadaWhat should readers look for when
statistically significant inconsistency is found in an NMA?
2. Potentially no NMA: Investigators may decide not to conduct a NMA in the presence
of excessive inconsistency. Direct and indirect results may be presented separately.
• The magnitude of the estimated inconsistency factor and its confidence interval may be
interpreted
3. Exploration of inconsistency: The network may be split into subgroups or may use
network meta-regression to account for differences across studies and comparisons.
4. Encompass inconsistency in the results: May apply DBT or LA models that relax
the consistency assumption
1. Assessment for data abstraction errors: Inconsistency in loops where a
comparison is informed by a single study is particularly suspicious for
data errors.
66
Knowledge Translation, Li Ka Shing Knowledge Institute, St. Michael's Hospital, Toronto, Canada
Analysis8. TASK: If both direct and indirect comparisons are available,
was agreement in treatment effects evaluated or discussed?
• If authors report that all but one comparison was not consistent, is the NMA still invalid? Would 1 inconsistent loop affect the whole network?
“According to the two-level random-effects model, no important incoherence between comparisons was detected (incoherence = .001). Only one particular comparison loop showed incoherence.”
Knowledge Translation, Li Ka Shing Knowledge Institute, St. Michael's Hospital, Toronto, Canada
Analysis9. TASK: In the presence of consistency between direct and
indirect comparisons, were both direct and indirect evidence included in the network meta-analysis?
Tricco et al BMC Medicine 2015
Network Estimates using the random effects model and common within-network heterogeneity
Ondansetron+DroperidolIVGranisetron+Dexamethasone
Ondansetron+MetoclopramideIVOndansetron+DexamethasoneDolasetron+Dexamethasone
Dolasetron+DroperidolIVGranisetronRamosetron
Granisetron+DroperidolIVTropisetron
OndansetronPalonosetron
DolasetronPalonosetron+Dexamethasone
0.14 (0.08,0.26) (0.06,0.36)0.15 (0.09,0.24) (0.06,0.35)
0.15 (0.06,0.42) (0.04,0.53)0.16 (0.12,0.23) (0.08,0.36)0.18 (0.06,0.49) (0.05,0.62)
0.19 (0.07,0.52) (0.06,0.65)0.26 (0.21,0.34) (0.12,0.56)0.28 (0.18,0.43) (0.12,0.64)
0.31 (0.11,0.82) (0.09,1.03)0.32 (0.23,0.43) (0.15,0.69)
0.36 (0.33,0.40) (0.18,0.74)0.38 (0.24,0.60) (0.16,0.89)
0.44 (0.30,0.63) (0.20,0.97)1.43 (0.20,10.14) (0.18,11.60)
OR 95%CI 95%PrITreatment effect
0 .2 1 3 12
Reference treatment: Placebo
Knowledge Translation, Li Ka Shing Knowledge Institute, St. Michael's Hospital, Toronto, Canada
Analysis10. With inconsistency or an imbalance in the distribution of
treatment effect modifiers across the different types of comparisons in the network of trials, did the researchers attempt to minimize this bias with the analysis?• Subgroup analyses, meta-regression, inconsistency models
Covariate (Treatment effect modifier, e.g., age)
Trea
tmen
t Eff
ect E
stim
ate
(e.g
., Lo
g-O
R)
Children Elderly
Knowledge Translation, Li Ka Shing Knowledge Institute, St. Michael's Hospital, Toronto, Canada
Analysis10. With inconsistency or an imbalance in the distribution of
treatment effect modifiers across the different types of comparisons in the network of trials, did the researchers attempt to minimize this bias with the analysis?• Subgroup analyses, meta-regression, inconsistency models• Apply an individual patient data NMA (IPD NMA)
Apply network meta-regression, assuming that the estimated relationship between effect modifier and treatment effect is not affected by other biases (e.g., aggregation bias ) – otherwise apply an IPD-NMA!
• IPD-NMA with patient-level covariates allows modeling within-study variation of effect modifiers. Hence, bias due to an imbalance of patient-level characteristics across comparisons can be minimized.
Knowledge Translation, Li Ka Shing Knowledge Institute, St. Michael's Hospital, Toronto, Canada
Analysis10. TASK: With inconsistency or an imbalance in the distribution of
treatment effect modifiers across the different types of comparisons in the network of trials, did the researchers attempt to minimize this bias with the analysis?• Subgroup analyses, meta-regression, inconsistency models• Apply an individual patient data NMA (IPD NMA)
Hint: This question should be answered with a ‘yes’: In the absence of inconsistency and absence of
differences in effect modifiers across treatment comparisons.
If meta-regression (or subgroup analyses) have been used to explore inconsistency or bias.o If inconsistency is identified and the authors did
not attempt to adjust, the answer should be ‘no’.
Knowledge Translation, Li Ka Shing Knowledge Institute, St. Michael's Hospital, Toronto, Canada
Analysis11. Was a valid rationale provided for the use of random-effects (RE)
or fixed-effect (FE) models?• Same considerations as in pairwise meta-analysis!
FE: no clinical or methodological heterogeneity is expected (Tricco, Open Med, 2012)
RE: clinical and/or methodological heterogeneity is expected (Tricco, CMAJ, 2014)
Knowledge Translation, Li Ka Shing Knowledge Institute, St. Michael's Hospital, Toronto, Canada
Analysis11. Was a valid rationale provided for the use of RE or FE models?
• FE = Fixed (common) effect, no effect modifiers
Random-effects model assumption:The observed study-specific effects estimate different true effects, which are related
and come from the same distribution
Fixed-effect model Assumption:Studies are sufficiently similar in aspects that could modify the treatment effect
There is a single true effect for all studies in the same treatment comparison.
Each study has each own true effect – all study-specific true effects are exchangeable
within the same treatment comparison.
Knowledge Translation, Li Ka Shing Knowledge Institute, St. Michael's Hospital, Toronto, Canada
Analysis11. Was a valid rationale provided for the use of RE or FE models?
FE and RE NMAs may be identical when the between-study variance is zero (if common within-network heterogeneity is assumed)
Fixed-effect model is often unrealistic “Since systematic reviews bring together studies that are diverse both clinically and
methodologically, heterogeneity in their results is to be expected.”
Random-effects meta-analysis suitable for unexplained small to moderate heterogeneity
When all treatment comparisons in a network include a single study, the RE model is not feasible
“We decided to apply a RE model, as we expected methodological and clinical heterogeneity across the included studies that compared the same
pairs of interventions.”
Tricco et al., BMC Med 2015
Higgins et al., BMJ 2003
Knowledge Translation, Li Ka Shing Knowledge Institute, St. Michael's Hospital, Toronto, Canada
Analysis11. TASK: Was a valid rationale provided for the use of RE or FE
models?Examples for rationale selecting between fixed-effect and random-effects model
“A random-effects network meta-analysis was conducted because we anticipated that the treatment effects were heterogeneous across the included RCTs. We assumed common heterogeneity across treatment comparisons, as the included treatments are of the same nature, hence, clinically reasonable to share a common heterogeneity parameter.”
“According to prior analyses in the field we anticipated a small number of included trials, which limits the applicability of random effect models because vague and weak informative prior distributions of the between-study variance have been shown to exert an unintentionally large degree of influence on any inference. Fixed effect models using vague prior distributions for all means were performed”
“Random effect models were consistently presented to account for the between study heterogeneity. However, in the case of the IPD analyses, fixed effect models were used given the limited number of studies to estimate the between study heterogeneity”
Knowledge Translation, Li Ka Shing Knowledge Institute, St. Michael's Hospital, Toronto, Canada
Analysis12. If a RE model was used, were assumptions about heterogeneity
explored or discussed?
• The heterogeneity variance can be estimated using several approaches– E.g., DerSimonian and Laird (DL), Maximum Likelihood (ML), Restricted Maximum Likelihood
(REML)
• In a Bayesian framework several priors can be assigned– E.g., Informative, Minimally informative, Vague
• Several heterogeneity assumptions can be considered– Common-within network heterogeneity: All treatment comparisons in the network are associated
with the same magnitude of heterogeneity
– Comparison-specific heterogeneity: Each treatment comparison in the network has its own amount of heterogeneity
– Common-within loop heterogeneity: The treatment comparisons included in each closed loop in the network share the same amount of heterogeneity.
Knowledge Translation, Li Ka Shing Knowledge Institute, St. Michael's Hospital, Toronto, Canada
12. TASK: If a RE model was used, were assumptions about heterogeneity explored or discussed?
Analysis
Examples for the heterogeneity assumption
“This model assumes that the degree of between-study within-comparison heterogeneity is constant across all intervention comparisons in the network.”
“We assume a common heterogeneity parameter across all comparisons”
“The between study variability of treatment effects was assigned a uniform (0, 2) prior density and assumed to be the same for all pairwise comparisons”
“The model was fitted into a Bayesian context with hierarchical models. The model used the assumption that the between-trial heterogeneity was equal across all comparisons.”
“We assumed common heterogeneity across treatment comparisons, as the included treatments are of the same nature and it was clinically reasonable to share a common heterogeneity parameter”
Knowledge Translation, Li Ka Shing Knowledge Institute, St. Michael's Hospital, Toronto, Canada
Analysis13. If there are indications of heterogeneity, were subgroup analyses or
meta-regression analyses with pre-specified covariates performed?• Pre-specified covariates based on patterns in the data
Placebo
BUDE
FLUT
MOMEAZD3199
FORMINDACSAML
VILA
ACLIGLYC
TIOT
UMEC
FORM/BECLO
FORM/BUDE
VILA/FLUT
SALM/FLUT
FORM/MOME
TIOT/FORMTIOT/SALMIND/TIOT
INDA/GLYCVILA/UMEC
GSK961081
TIOT/FLUT/SALM
TIOT/BUDE/FORM
Full network Purple: No exacerbations in previous year; Red: Had an exacerbation in previous year; Green: Not Reported
Placebo
BUDE
FLUT
MOME
AZD3199FORM
INDACSAMLVILA
ACLIGLYC
TIOT
UMEC
FORM/BECLO
FORM/BUDE
VILA/FLUT
SALM/FLUT
FORM/MOMETIOT/FORM
TIOT/SALM IND/TIOTINDA/GLYC
VILA/UMEC
GSK961081
TIOT/FLUT/SALM
TIOT/BUDE/FORM
Knowledge Translation, Li Ka Shing Knowledge Institute, St. Michael's Hospital, Toronto, Canada
Analysis13. If there are indications of heterogeneity, were subgroup analyses or
meta-regression analyses with pre-specified covariates performed?• Pre-specified covariates based on patterns in the data
Sub-network based on patients with exacerbations in past year or more
Placebo
BUDE
FLUT
FORM
INDACSAML
VILA
GLYC
TIOT
FORM/BECLO
FORM/BUDE
VILA/FLUT
SALM/FLUT TIOT/SALMINDA/GLYC
TIOT/FLUT/SALM
TIOT/BUDE/FORM
Knowledge Translation, Li Ka Shing Knowledge Institute, St. Michael's Hospital, Toronto, Canada
Analysis13. TASK: If there are indications of heterogeneity, were subgroup analyses
or meta-regression analyses with pre-specified covariates performed?
Knowledge Translation, Li Ka Shing Knowledge Institute, St. Michael's Hospital, Toronto, Canada
Validity [or Credibility as per ISPOR]
Knowledge Translation, Li Ka Shing Knowledge Institute, St. Michael's Hospital, Toronto, Canada
Reporting quality14. TASK: Is a graphical or tabular representation of the
evidence network provided with information on the number of RCTs per direct comparison?• Network diagram, table with studies in the rows and interventions and
results in the columns
NPH [od / td]
NPH [od]
D[od/ td]
D[fd]
D[od]
G [td]
G[od]
6
4
2
1
11
13
Comparison # Studies # Patients # EventsA vs B 2 185 94A vs C 4 708 428A vs D 3 586 283B vs C 1 102 70B cs D 2 1087 592C vs D 1 300 28
Knowledge Translation, Li Ka Shing Knowledge Institute, St. Michael's Hospital, Toronto, Canada
Reporting quality15. TASK: Are the individual study results reported?
• Increases reproducibility and face validity
Treatment A B C D E
Study# of
Events# of
Patients# of
Events# of
Patients# of
Events# of
Patients# of
Events# of
Patients# of
Events# of
Patients1 4 50 37 50 2 1 35 20 45 3 1 17 18 34 52 68 4 6 158 141 158 5 8 267 437 504 20 3306 88 112 67 120 7 0 57 15 70 8 3 27 4 30 9 13 150 15 140
10 0 49 20 70 11 15 71 2 80 12 0 87 128 181 13 0 89 136 178
Knowledge Translation, Li Ka Shing Knowledge Institute, St. Michael's Hospital, Toronto, Canada
Reporting quality16. TASK: Are results of direct comparisons reported separately
from results of the indirect comparisons or NMA?• Consistency assumption
Tricco et al BMJ 2014
Knowledge Translation, Li Ka Shing Knowledge Institute, St. Michael's Hospital, Toronto, Canada
Reporting quality17. TASK: Are all pairwise contrasts between interventions as
obtained with the NMA reported along with measures of uncertainty?• For example, ORs and 95% CrIs
A
3.71(0.65 – 6.00) B
1.83(0.13 – 3.00)
1.66(0.19 – 4.76) C
2.70(0.45 –5.00)
2.54(0.18 – 5.32)
1.52(0.09 – 7.72) D
League Table
Tricco et al BMJ 2014
Knowledge Translation, Li Ka Shing Knowledge Institute, St. Michael's Hospital, Toronto, Canada
Reporting quality18. Is a ranking of interventions provided given the reported
treatment effects and its uncertainty by outcome?• Ranking probabilities
Tricco et al BMC Medicine 2015
Placebo
O
G
DTO+D
P
R
O+D_IV
O+M_IV
G+Dex
P+Dex
D+Dex
D+D_IV
G+D_IV
Knowledge Translation, Li Ka Shing Knowledge Institute, St. Michael's Hospital, Toronto, Canada
Reporting quality18. Is a ranking of interventions provided given the reported
treatment effects and its uncertainty by outcome?• Ranking probabilities
0.2
.4.6
.81
0.2
.4.6
.81
0.2
.4.6
.81
0.2
.4.6
.81
0.2
.4.6
.81
0.2
.4.6
.81
0.2
.4.6
.81
0.2
.4.6
.81
0.2
.4.6
.81
0.2
.4.6
.81
0.2
.4.6
.81
0.2
.4.6
.81
0.2
.4.6
.81
0.2
.4.6
.81
0.2
.4.6
.81
1 3 5 7 9 11 13 15 1 3 5 7 9 11 13 15 1 3 5 7 9 11 13 15 1 3 5 7 9 11 13 15
1 3 5 7 9 11 13 15 1 3 5 7 9 11 13 15 1 3 5 7 9 11 13 15 1 3 5 7 9 11 13 15
1 3 5 7 9 11 13 15 1 3 5 7 9 11 13 15 1 3 5 7 9 11 13 15 1 3 5 7 9 11 13 15
1 3 5 7 9 11 13 15 1 3 5 7 9 11 13 15 1 3 5 7 9 11 13 15
Dolasetron Dolasetron+Dexamethasone Dolasetron+DroperidolIV Granisetron
Granisetron+Dexamethasone Granisetron+DroperidolIV Ondansetron Ondansetron+Dexamethasone
Ondansetron+DroperidolIV Ondansetron+MetoclopramideIV Palonosetron Palonosetron+Dexamethasone
Placebo Ramosetron Tropisetron
Pro
babi
litie
s of
eac
h ra
nk
Rank
Treatment SUCRA Mean Rank
Ond+Drop IV 85.4 3Gran+Dex 84 3
Ond+Dex 79.5 4Ond+Met IV 78.6 4
Dol+Dex 71.8 5Dola+Drop IV 68.2 6
Gran 54.4 7Ramos 49.2 8Tropis 41.6 9
Gran+Drop IV 44.6 9Ond 30.8 11
Palon 29.1 11 Dolas 20.8 12
Placebo 4.6 14Palon+Dex 7.5 14
Tricco et al BMC Medicine 2015
Knowledge Translation, Li Ka Shing Knowledge Institute, St. Michael's Hospital, Toronto, Canada
Reporting quality18. Is a ranking of interventions provided given the reported
treatment effects and its uncertainty by outcome?• Rank-heat plot
Veroniki et al JCE 2016
Knowledge Translation, Li Ka Shing Knowledge Institute, St. Michael's Hospital, Toronto, Canada
Reporting quality18. TASK: Is a ranking of interventions provided given the
reported treatment effects and its uncertainty by outcome?
Knowledge Translation, Li Ka Shing Knowledge Institute, St. Michael's Hospital, Toronto, Canada
Reporting quality19.Is the effect of important patient characteristics on
treatment effects reported?• Relative treatment effects for different levels of the effect modifier
Author, year
Sample size
Age Category
% female ASA statusSurgery
type
H/o motion
sickness H/o PONV
Comorbidities
Randomized clinical trials (n=429)
Browning, 2013
118 Adults NR I or II
Obstetrics &
Gynaecological
NR NRMental health
Aleyasin, 2012
120 Adults 100 NR Breast NR NR NR
Blitz, 2012 118 Adults 84 I or IIMiscellaneo
usNR NR NR
Choi, 2012 120 Adults 100 I or IIOrthopaedi
cYES YES NR
de Orange, 2012 129 Children 24 I or II
Miscellaneous
NR YES NR
Eidi, 2012 219 Adults 72 I or IIEar, nose
and larynxNR NR NR
Tricco et al BMC Medicine 2015
Knowledge Translation, Li Ka Shing Knowledge Institute, St. Michael's Hospital, Toronto, Canada
Reporting quality19. TASK: Is the effect of important patient characteristics on
treatment effects reported?• Other important items for reporting can be found in the PRISMA for
NMA
Hutton et al Annals of Int Med 2015
Knowledge Translation, Li Ka Shing Knowledge Institute, St. Michael's Hospital, Toronto, Canada
Validity [or Credibility as per ISPOR]
Knowledge Translation, Li Ka Shing Knowledge Institute, St. Michael's Hospital, Toronto, Canada
Interpretation20. Are the conclusions fair and balanced?
• Consistent with NMA results and available evidence base, credible analysis methods, no bias concerns
NPH [od / td]
NPH [od]
D[od/ td]
D[fd]
D[od]
G [td]
G[od]
6
4
2
1
11
1
Network of hypoglycemia
Tricco et al BMJ 2014
Inconsistency (DBT model): χ2=7, d.f.=1, P-value=0.0082
Should we trust these results?
Knowledge Translation, Li Ka Shing Knowledge Institute, St. Michael's Hospital, Toronto, Canada
Interpretation20. Are the conclusions fair and balanced?
• Consistent with NMA results and available evidence base, credible analysis methods, no bias concerns
NPH [od / td]
D[od/ td]
D[fd]
D[od]
G [td]
G[od]
6
31
11
1
Network of hypoglycemia
Tricco et al BMJ 2014
Inconsistency (DBT model): χ2=2.56, d.f.=1, P-value=0.1093
High risk of bias in 3 studies caused inconsistency!
Knowledge Translation, Li Ka Shing Knowledge Institute, St. Michael's Hospital, Toronto, Canada
Interpretation20. TASK: Are the conclusions fair and balanced?
Knowledge Translation, Li Ka Shing Knowledge Institute, St. Michael's Hospital, Toronto, Canada
Validity [or Credibility as per ISPOR]
Knowledge Translation, Li Ka Shing Knowledge Institute, St. Michael's Hospital, Toronto, Canada
Conflict of interest21. Were there any potential conflicts of interest?
• Financial or personal relationships or affiliations that could bias the results
Knowledge Translation, Li Ka Shing Knowledge Institute, St. Michael's Hospital, Toronto, Canada
Conflict of interest21. Were there any potential conflicts of interest?
• Financial or personal relationships or affiliations that could bias the results
Knowledge Translation, Li Ka Shing Knowledge Institute, St. Michael's Hospital, Toronto, Canada
Conflict of interest21. TASK: Were there any potential conflicts of interest?
Knowledge Translation, Li Ka Shing Knowledge Institute, St. Michael's Hospital, Toronto, Canada
Conflict of interest22. TASK: If yes, were steps taken to address these?
• All COI should be noted, publication should be peer-reviewed, contribution of each author reported
Tricco et al BMJ 2014
Knowledge Translation, Li Ka Shing Knowledge Institute, St. Michael's Hospital, Toronto, CanadaExample – NMA published in BMJ
OpenHow would you explain the results to a decision maker?
Tricco et al BMJ Open 2015
Knowledge Translation, Li Ka Shing Knowledge Institute, St. Michael's Hospital, Toronto, Canada
1. Baldwin D, Woods R, Lawson R, Taylor D. Efficacy of drug treatments for generalised anxiety disorder: systematic review and meta-analysis. BMJ. 2011 Mar 11;342:d1199.
2. Bucher HC, Guyatt GH, Griffith LE, Walter SD. The results of direct and indirect treatment comparisons in meta-analysis of randomized controlled trials. J Clin Epidemiol 1997;50(6):683-91.
3. Chaimani A, Higgins JP, Mavridis D, Spyridonos P, Salanti G. Graphical tools for network meta-analysis in STATA. PLoS One. 2013 Oct 3;8(10):e76654
4. Cipriani A, Higgins JP, Geddes JR, Salanti G. Conceptual and Technical Challenges in Network Meta-analysis. Ann Intern Med 2013;159:130-137.
5. Donegan S, Williamson P, D'Alessandro U, Tudur-Smith C. Assessing key assumptions of network meta-analysis: a review of methods Res Synth Meth. 2013; 4(4), 291-323.
6. Higgins JPT, Green S (editors). Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0 [updated March 2011]. The Cochrane Collaboration, 2011. Available from www.cochrane-handbook.org.
7. Jansen J, Trikalinos TA, Cappelleri JP, et al, Indirect treatment comparison/network meta-analysis study questionnaire to assess study relevance and credibility to inform health care decision-making: an ISPOR-AMCP-NPC Good Practice Task Force report. Value Health 2014;17, 157–173
8. Lu G, Ades A. Modeling between-trial variance structure in mixed treatment comparisons. Biostatistics. 2009; 10(4) :792-805.
9. Moher D, Liberati A, Tetzlaff J, Altman DG; PRISMA Group. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. BMJ. 2009 Jul 21;339:b2535.
10. Nikolakopoulou A, Chaimani A, Veroniki AA, Vasiliadis HS, Schmid CH, Salanti G, Characteristics of networks of interventions: a description of a database of 186 published networks. PLoS One. 2014;9(1):e86754.
11. Higgins JPT, Jackson D, Barrett JK, Lu G, Ades AE, White IR. Consistency and inconstency in network meta-analysis: concepts and models for multi-arm studies. Research Synth. Meth. 2012;3, 98-110.
12. Salanti G. Indirect and mixed-treatment comparison, network, or multiple-treatments meta-analysis: many names, many benefits, many concerns for the next generation evidence synthesis tool. Res. Synth. Meth. 2012;3(2), 80-97.
13. Shamseer L, Moher D, Clarke M, Ghersi D, Liberati A, et al; PRISMA-P Group. Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015: elaboration and explanation. BMJ. 2015 Jan 2;349:g7647.
References (1)
Knowledge Translation, Li Ka Shing Knowledge Institute, St. Michael's Hospital, Toronto, Canada
14. Sampson M, McGowan J, Cogo E, Grimshaw J, Moher D, Lefebvre C. An evidence-based practice guideline for the peer review of electronic search strategies. J Clin Epidemiol. 2009 Sep;62(9):944-52.
15. Song F, Xiong T, Parekh-Bhurke S, Loke YK, Sutton AJ, Eastwood AJ, et al. Inconsistency between direct and indirect comparisons of competing interventions: Meta-epidemiological study. BMJ. 2011;343:d4909.
16. Tricco AC, Ashoor HM, Antony J, Beyene J, Veroniki AA, Isaranuwatchai W, et al. Safety, effectiveness, and cost effectiveness of long acting versus intermediate acting insulin for patients with type 1 diabetes: systematic review and network meta-analysis, BMJ. 2014;349:g5459
17. Tricco AC, Soobiah C, Blondal E, Veroniki AA, Khan PA, Vafaei A, et al. Comparative efficacy of serotonin (5-HT3) receptor antagonists in patients undergoing surgery: A systematic review and network meta-analysis, BMC Medicine. 2015; 13:136
18. Tricco AC, Alateeq A, Tashkandi M, Mamdani M, Al-Omran M, Straus SE. Histamine H2 receptor antagonists for decreasing gastrointestinal harms in adults using acetylsalicylic acid: systematic review and meta-analysis. Open Med. 2012 Aug 21;6(3):e109-17.
19. Tricco AC, Antony J, Ivers NM, Ashoor HM, Khan PA, Blondal E, Ghassemi M, MacDonald H, Chen MH, Ezer LK, Straus SE. Effectiveness of quality improvement strategies for coordination of care to reduce use of health care services: a systematic review and meta-analysis. CMAJ. 2014 Oct 21;186(15):E568-78
20. Tricco AC, Soobiah C, Ho JM, Berliner S, Ng CH, Hamid JS, Qi Y, Perrier L, Hemmelgarn B, Majumdar SR, Straus SE. Comparative safety and effectiveness of cognitive enhancers for Alzheimer's dementia: A systematic review and network meta-analysis. Submitted to Annals of Internal Medicine in 2015.
21. Veroniki AA, Vasiliadis HS, Higgins JP, Salanti G. Evaluation of inconsistency in networks of interventions. Int J Epidemiol 2013;42, 332-345.
22. Veroniki AA, Mavridis D, Higgins JP, Salanti G. Statistical evaluation of inconsistency in a loop of evidence: a simulation study informed by empirical evidence. BMC Med. Res. Methodol. 2014; 14, 106.
23. Veroniki AA, Straus SE, Fyraridis A, Tricco AC. The rank-heat plot is a novel way to present the results from a network meta-analysis including multiple outcomes. J. Clin. Epidemiol. 2016; pii: S0895-4356(16)00153-0.
24. White IR, Barret JK, Jackson D, Higgins JPT. Consistency and inconsistency in multiple treatments meta-analysis: model estimation using multivariate meta-regression. Research Synthesis Methods 2012;3(2):111-25
References (2)
Knowledge Translation, Li Ka Shing Knowledge Institute, St. Michael's Hospital, Toronto, Canada