THE USE OF SYSTEMATIC REVIEWS IN EVIDENCE-BASED

DECISION MODELLING

Nicola Cooper, Alex Sutton,

Keith Abrams, Paul LambertDepartment of Epidemiology & Public Health,

University of Leicester.

• Increasingly decision models are being developed to inform complex clinical/economic decisions

• Parameters can include: –clinical effectiveness, –costs, –disease progression rates, and –utilities

• Evidence based - use systematic methods for evidence synthesis to estimate model parameters with appropriate levels of uncertainty

                                     BACKGROUND

                                     SOURCES OF UNCERTAINTY IN

DECISION MODELS

• Statistical error

• Systematic error

• Evidence relating to parameters indirectly

• Data quality, publication bias, etc.

                                     

Stable

 Progressive

Death

                           

   

MARKOV MODEL – TAXANE vs. STANDARD(2nd line treatment of advanced breast cancer)

Response

Cycle length 3 weeks

QR , CR QS , CS

QP , CP

QD = 0

Quality of Life (Q)

Cost (C)

PSR

PSP

PPD

PRP

PR PS

PP

Probability (P)

                                     

1. Meta-analyse available evidence to obtain a distribution for each model parameter using random effect models

2. Transform the pooled results, if necessary, and input into the model directly as a distribution and evaluate the model

3. All analyses (decision model and subsidiary analyses) implemented in one cohesive statistical model/program

4. Implemented in a fully Bayesian way using Markov Chain Monte Carlo simulation within WinBUGS software

5. All prior distributions intended to be ‘vague’. Where uncertainty exists in the value of parameters (i.e. most of them!) they are treated as random variables

GENERAL APPROACH

                                     

Stable

 Progressive

Death

                           

   MARKOV MODEL – TAXANE vs. STANDARD

(2nd line treatment of advanced breast cancer)

Response

Cycle length 3 weeks

QR , CR QS , CS

QP , CP

QD = 0

Quality of Life (Q)

Cost (C)

PSR

PSP

PPD

PRP

PRPS

PP

Probability (P)

1) M-A of RCTs: Annual ln(odds) of responding

Odds - log scale.1 .25 1 5

Combined

Bonneterre

Sjostrom

Nabholtz

Chan

                                     

MODEL PARAMETER ESTIMATION e.g. PSR, TAX – The probability of moving from stable to

response in a 3 week period

mu.rsprtD sample: 12001

-5.0 0.0 5.0

0.0 0.5 1.0 1.5 2.0

2) Pooled ln(odds) distribution

3) Transformation of ln(odds)distrn to transition probability

)3/52/(1

/1

]42.01[1

)],(1[1

jjo ttP

4) Apply to model

Respond

Stable

Progressive

Death

-0.3 (-0.9 to 0.3) PSR

THE REMAINING PARAMETERS

• The Transition Probabilities need estimating for each intervention being compared

• Costs and Utilities can be extracted from the literature and synthesised using a similar approach within the same framework

                                     META-ANALYSES OF LITERATURE

(where required) No. of

studies Time in weeks

(95% Credible Interval) Progression-free time 3 25 (15 to 24)

Time to response from stable 1 12 (6 to 18) Time to progressive from response 1 35 (29 to 41)

Overall survival time 3 53 (35 to 74) Probabilities

Response rate 4 0.43 (0.29 to 0.58) % moving directly to progressive at stage 2. 1 0.13 (0.08 to 0.18)

% with infections / febrile neutropenia 3 0.18 (0.04 to 0.56) % hospitalised with infection / febrile neutropenia 1 0.08 (0.05 to 0.11)

% dying from infections / febrile neutropenia 1 0.01 (0.00 to 0.02) % discontinue treatment due to adverse event 3 0.16 (0.03 to 0.49)

% with Neutropenia grades 3 & 4 2 0.94 (0.82 to 0.98) % with Anaemia grades 3 & 4 2 0.03 (0.00 to 0.28) % with Diarrhoea grades 3 & 4 3 0.09 (0.06 to 0.14) % with Stomatis grades 3 & 4 3 0.08 (0.04 to 0.14) % with vomiting grades 3 & 4 2 0.03 (0.00 to 0.12)

% with fluid retention grades 3 & 4 3 0.05 (0.02 to 0.12) % with cardiac toxicity grades 3 & 4 1 0.00 (0.00 to 0.02)

                                     TRANSITION PROBABILITIES FOR MODEL

(Derived from M-As)

Transition Probabilities (95% Credible Interval)

Infection/FN 0.09 (0.02 to 0.32)

Hospitalised due to infection/FN 0.04 (0.03 to 0.05)

Dying from infection/FN after hospitalisation 0.00 (0.00 to 0.01)

Discontinuation due to major adverse events 0.04 (0.04 to 0.16)

Adverse events – Neutropenia 0.50 (0.34 to 0.63)

Adverse events – Anaemia 0.01 (0.00 to 0.07)

Adverse events – Diarrhoea 0.02 (0.01 to 0.37)

Adverse events – Stomatis 0.02 (0.01 to 0.04)

Adverse events – Vomiting 0.01 (0.00 to 0.03)

Adverse events – Fluid retention 0.01 (0.00 to 0.03)

Adverse events – Cardiac toxicity 0.00 (0.00 to 0.01)

Transition directly to ‘progressive’ state 0.12 (0.08 to 0.18)

Transition ‘stable’ to ‘stable’ 0.65 (0.44 to 0.75)

Transition ‘stable’ to ‘response’ 0.16 (0.11 to 0.28)

Transition ‘stable’ to ‘progressive’ 0.18 (0.11 to 0.37)

Transition ‘response’ to ‘response’ 0.94 (0.93 to 0.95)

Transition ‘response’ to ‘progressive’ 0.06 (0.05 to 0.07)

Transition ‘progressive’ to ‘progressive’ 0.93 (0.79 to 0.96)

Transition ‘progressive’ to ‘death’ 0.07 (0.04 to 0.21)

                                     EVALUATION OF THE MODEL

• A cohort of 1,000 persons is run through the model over 35 3-weekly cycles (until the majority of people are dead) for each treatment option

• Costs and utilities are calculated at the end of each cycle and the average cost and utilities for an individual across all 35 cycles for each treatment option are calculated

• This process is repeated 4,000 times (each time different values from each parameter distribution are sampled)

Bayesian (MCMC) Simulations

-£4,000

-£2,000

£0

£2,000

£4,000

£6,000

£8,000

£10,000

-0.50 -0.40 -0.30 -0.20 -0.10 0.00 0.10 0.20 0.30 0.40 0.50

Incremental utility

Inc

rem

en

tal

co

st

Standard dominates

Taxane more effective but more costly

Taxane less costly but less

effective

Taxanedominates

                                     COST-EFFECTIVENESS PLANE

• Evidence that post MI, the risk of a stroke is reduced in patients with atrial fibrillation by taking warfarin

• However, there is a risk of a fatal hemorrhage as a result of taking warfarin

• For whom do the benefits outweigh the risks?

                                     CLINICAL NET BENEFIT

- Warfarin for non-rheumatic atrial fibrillation

EVALUATION OF NET BENEFIT

(Risk of stroke Relative reduction in risk of stroke)

- (Risk of fatal bleed Outcome ratio)

=

Net Benefit

EVALUATION OF NET BENEFIT

(Risk of stroke Relative reduction in risk of stroke)

- (Risk of fatal bleed Outcome ratio)

=

Net Benefit

Multivariate riskequations

Meta analysisof RCTs

Meta analysis of RCTs obs studies QoL study

EVALUATION OF NET BENEFIT

(Risk of stroke Relative reduction in risk of stroke)

- (Risk of fatal bleed Outcome ratio)

=

Net Benefit

Multivariate riskequations

Meta analysisof RCTs

Meta analysis of RCTs obs studies QoL study

0.002 0.004 0.006 0.008 0.010 0.012 0.014

050

100

150

200

250

300

risk of bleed per year

-2.95 -2.90 -2.85 -2.80 -2.75 -2.70 -2.65

02

46

810

-1.5 -1.0 -0.5 0.0 0.5 1.0

02

46

reduction in relative risk

0 20 40 60 80 100

0.0

0.1

0.2

0.3

0.4

Outcome ratio

Multivariate Risk Equation Data Net Benefit (measured in stroke equivalents)

% of

cohort

T hrombo -

embolism

rate (%

per year

(95% CI))

Mean

(s.e.)

Median

(95%

CrI)

Probability of

Benefit > 0

Simulated PDF

12

17.6 (10.5

to 29.9)

- 0.0004

(0.15)

0.06

( - 0.29 to

0.20)

54.2 %

-0.8 -0.6 -0.4 -0.2 0.0 0.2 0.4

01

23

45

6

2 or 3 Clinical factors

EVALUATION OF NET BENEFIT

(Risk of stroke Relative reduction in risk of stroke)

- (Risk of fatal bleed Outcome ratio)

=

Net Benefit

Multivariate riskequations

Meta analysisof RCTs

Meta analysis of RCTs obs studies QoL study

0.002 0.004 0.006 0.008 0.010 0.012 0.014

050

100

150

200

250

300

risk of bleed per year

-2.95 -2.90 -2.85 -2.80 -2.75 -2.70 -2.65

02

46

810

-1.5 -1.0 -0.5 0.0 0.5 1.0

02

46

reduction in relative risk

0 20 40 60 80 100

0.0

0.1

0.2

0.3

0.4

Outcome ratio

ADVANTAGES OF APPROACH Synthesis of evidence, transformation of variables &

evaluation of a complex Markov model carried out in a unified framework

Facilitates sensitivity analysis

Provides a framework to incorporate prior beliefs of experts

Allows for correlation induced where studies included in the estimation of more than one parameter

Uncertainty in all model parameters automatically taken into account

Rare event data modelled ‘exactly’ (i.e. removes the need for continuity corrections) & asymmetry in posterior distribution propagated

FURTHER ISSUES1. Handling indirect comparisons correctly

• E.g. Want to compare A vs. C but evidence only available on A vs. B & B vs. C etc.

• Avoid breaking randomisation

2. Necessary complexity of model?• When to use the different approaches outlined

above?

3. Incorporation of Expected Value of (Perfect/Sample) Information

4. Incorporation of all uncertainties

1. Handling indirect comparisons correctly• E.g. Want to compare A vs. C but evidence only

available on A vs. B & B vs. C etc.

• Avoid breaking randomisation

2. Necessary complexity of model?• When to use the different approaches outlined

above?

3. Incorporation of Expected Value of (Perfect/Sample) Information

4. Incorporation of all uncertainties