Value of Information for Complex Economic Models

Jeremy Oakley

Department of Probability and Statistics,University of Sheffield.

Paper available from www.sheffield.ac.uk/chebs/papers.html

Outline

1. Motivation

2. Expected value of perfect information (EVPI)

3. Emulators and Gaussian processes

4. Illustration: GERD model

1) Introduction

• An economic model is to be used to predict the cost-effectiveness of a particular treatment(s).

• The economic model will require the specification of various input parameters. Values of some or all of these are uncertain.

• This implies the output of the model, the cost-effectiveness of the treatment is also uncertain.

Introduction

• We wish to identify which input parameters are the most influential in driving this output uncertainty.

• Should we learn more about these parameters before making a decision?

Introduction

• A measure of importance for an input variable have been proposed, based on the expected value of perfect information (EVPI) (Felli and Hazen, 1998, Claxton 1999).

• Computing the values of these measures is conventionally done using Monte Carlo techniques. These invariably require a very large numbers of runs of the economic model.

Introduction

• For computationally expensive models, this can be completely impractical.

• We present an efficient alternative to Monte Carlo, in terms of the number of model runs required.

2) EVPI

• We work with net benefit: the monetary value or utility of a treatment is

K x efficacy – cost with K the monetary value of a unit

increase in efficacy. • The net benefit of any treatment option

will be a function of the parameters in the economic model.

EVPI

• Denote the net benefit of treatment option t given model parameters X to be

NB (t , X )• Given X, the economic model returns

NB (t , X ) for each t . • The ‘true’ values of the model

parameters X are uncertain.

EVPI

• The baseline decision is to choose t with the largest expected net benefit:

NB* = maxt EX {NB (t , X )}

• The decision maker will have utility NB* if they choose the best treatment now with no additional information.

EVPI

• Now suppose the decision-maker chooses to learn the value of all the uncertain input variables X before choosing a treatment.

• They would then choose the treatment with the highest net benefit conditional on X, i.e., they would consider

maxt {NB (t , X )}

EVPI

• Before actually observing X, they will expect to achieve a net benefit of

EX [maxt {NB (t , X )}]

• The expected value of this course of action is the expected gain in net benefit over the baseline decision:

EX [maxt {NB (t , X )}] – NB*.

• This is the (global) EVPI.

Partial EVPI

• Now suppose the decision-maker chooses to learn the value of a single uncertain input variable Y , an element of X before making a decision.

• They would then choose the treatment with the highest net benefit conditional on Y , i.e., they would consider

maxt EX | Y {NB (t , X )}

Partial EVPI

• The expected value of learning Y before Y

is actually observed is then:EY [maxt EX |Y {NB (t , X )}] – NB *

• This is the partial expected value of perfect information (partial EVPI) for Y .

• The partial EVPI is zero if the decision-maker would choose the same treatment for any (plausible) value of Y .

Computing partial EVPIs

• We need to evaluateEY [maxt EX |Y {NB (t , X )}]

for each element Y in X.• The outer expectation EY is a one-

dimensional integral, and can be evaluated using numerical integration.

• The term maxt EX |Y is the maximum of (several) higher-dimensional integrals. This requires a large Monte Carlo sample to be evaluated.

Patient Simulation Models

• Computing partial EVPIs for computationally cheap models, while not trivial, is relatively straightforward.

• However, for one class of models, patient simulation models, a sensitivity analysis using Monte Carlo methods will be out of reach for the model user.

Patient Simulation Models

• An example is given in Kanis et al (2002) for modelling osteoporosis:

• For an osteoporosis patient, a bone fracture significantly increases the risk of a subsequent fracture.

• Residential status of a patient needs to be tracked, in order that costs are not double-counted.

Patient Simulation Models

• Progress is to be modelled over a 10 year period. Including the approptiate features in the model necessitates a patient simulation approach.

• The net benefit for a given set of input parameters is obtained by sampling events for a large number of patients.

• The model takes over an hour for a single run at one set of input parameters.

Patient Simulation Models

• For a model with 20 uncertain input variables, computing the partial EVPI reliably using Monte Carlo for each input variable would require a possible minimum of 500,000 model runs.

• At one hour for each run, this would take 57 years!

• Something more efficient is needed…

3) Emulators

• For each treatment option t, and given values for the input parameters X = x, the economic model returns NB (t , x )

• We think of the model as a collection of functions

NB (t , x ) = ft (x)

• Partial EVPIs can be computed more efficiently by exploiting the `smoothness’ of each ft (x)

Emulators

• We can compute partial EVPIs more efficiently through the use of an emulator.

• An emulator is a statistical model of the original economic model which can then be used as a fast approximation to the model itself.

• An approach used by Sacks et al (1989) for dealing with computationally expensive computer models.

Gaussian processes

• Any regression technique can be used. We employ a nonparametric regression technique based on Gaussian processes (O’Hagan, 1978).

• The gaussian process model for the function ft (x) is non-parametric; the only assumption made about ft (x) is that it is a continuous function.

Gaussian processes

• In the Gaussian process model, ft (x) is thought of as an unknown function, and uncertainty about ft (x) is described by a normal distribution.

• Correlation between ft (x1) and ft (x2) is modelled parametrically as a function of ||x1-x2||

Gaussian processes

• The partial EVPI for input variable Y is given by

EY [maxt EX |Y {NB (t , X )}] – NB *

• We need to evaluate EX |Y {NB (t , X )} for each t at various values of Y.

• Denote G (X |Y) to be the distribution of X given Y. Then

EX |Y {NB (t , X )} = ft (x) dG (x |y )

Gaussian processes

• We can use Bayesian quadrature (O’Hagan, 1993) to rapidly speed up the computation:

• Under the Gaussian process model for ft

(x), ft (x) dG (x |y )

has a normal distribution, and can be evaluated (almost) instantaneously.

• This reduces the number of model runs required from 100,000s to 100s.

4) Example: GERD model

• The GERD model, presented in O’Brien et al (1999) predicts the cost-effectiveness of a range of treatment strategies for gastroesophageal reflux disease.

• Various uncertain inputs in the model related to treatment efficacies, resource uses by patients.

• Model outputs mean number of weeks free of GERD symptoms, and mean cost of treatment for a particular strategy.

Example: GERD model

• We consider a choice between three treatment strategies:› Acute treatment with proton pump inhibitors

(PPIs) for 8 weeks, then continuous maintenance treatment with PPIs at the same dose.

› Acute treatment with PPIs for 8 weeks, then continuous maintenance treatment with hydrogen receptor antagonists (H2RAs).

› Acute treatment with PPIs for 8 weeks, then continuous maintenance treatment with PPIs at the a lower dose.

Example: GERD model

• There are 23 uncertain input variables. • Distributions for uncertain inputs detailed in

Briggs et al (2002).• We estimate the partial EVPI for each input

variable, based on 600 runs of the GERD model.

• We assume a value of $250 for each week free of GERD symptoms. (It is straightforward to repeat our analysis for alternative values).

Example: GERD model

Conclusions.

• The use of the Gaussian process emulator allows partial EVPIs to be computed considerably more efficiently.

• Sensitivity analysis feasible for computationally expensive models.

• Can also be extended to value of sample information calculations.