Deciding how to decide

Multi-Criteria Decision Analysisis (probably) the future for

both Health Technology Assessment

andShared Clinical Decision Making

Jack DowieProfessor Emeritus of Health Impact Analysis

London School of Hygiene and Tropical Medicinejack.dowie@lshtm.ac.uk

Abstract• I introduce JUDEMAKIA, a map of the world of Judgement and

Decision Making, in which we can locate various Belief Technologies, Preference Technologies and Decision Technologies in the framework provided Ken Hammond’s Cognitive Continuum of changing Analysis to Intuition balances.

• I locate (Single-Criterion) Decision Analysis – SCDA - and particular implementations of Multi-Criteria Decision Analysis – MCDA (Analytic Hierarchy Process, Annalisa) on this map

• I introduce some applications of MCDA in both the health service context (Health Technology Assessment - HTA) and clinical context (Shared Clinical Decision Making - SCDM)

• I suggest that Decision Resource-Effectiveness Analysis – DREA - is the appropriate way to tackle the meta decision of how to decide to decide – i.e. determine which Decision Technology should be used for a specific decision in a specific context, given the relative weights assigned to the high level criteria of rigour and practicality

• DREA is best implemented via MCDA

Some of the slides in the handout may not appear in the presentation but they have been included for later follow-up if desired.

Taxonomy of tasks and technologies• To produce beliefs*, elicit beliefs and evaluate belief

claims we need a Belief* Technology [*Knowledge/Evidence]

• To establish what preferences are held and elicit preference judgements we need a

Preference* Technology [*Value]• To make a decision we need a

Decision TechnologyDTs need inputs from BTs and PTs

and BTs and PTs need inputs from DTsand the transfer process requires ITs and CTs

• To transfer information we need anInformation Technology

• To communicate we need aCommunication Technology

Examples• Belief or Knowledge Technologies

judgement, interview, panel, cohort study, RCT• Preference or Values Technologies

judgement, interview, Visual Analog Scale, Standard Gamble, Time Trade-Off, DCE

• Decision Technologiesjudgement, coin toss, meeting, pro/con checklist, F&F decision analysis, Comp Stoch Decision Model

• Information / Communication Technologiesbody language, ppt presentation, Report with tables and graphs, Youtube video, twitter

A very inefficient BT- Gary Larson

A very simple PT– Randy Glasbergen

A moderately analytic DT

3 Risk factors

Black

Male

Baseball capbackwards

4 Mitigating factors

White

Groceries

Humming Motown

Over 40

3 Risk factors

White

Male

Wrong neighbourhood

4 Mitigating factors

Loafers

Fed Ex envelope

Whistling Sondheim

Polo shirt

– Garry Trudeau

A fairly inappropriate ICT– Nick D Kim

2005 2006

Underlying framework• Cognitive Continuum Theory of Ken Hammond

Human Judgment and Social Policy: Irreducible Uncertainty, Inevitable Error, Unavoidable Injustice (Oxford UP, 1996)

• 2 dimensions: • COGNITIVE dimension (how we think about a JDM task)

which runs from highly Intuitively to highly Analytically• TASK STRUCTURE dimension (how a JDM task presents itself to us)

which runs from Very well-structured to Very ill-structured

• Any instantiation of a BT, PT, DT or I/CT can be• characterised by its Analysis-to-Intuition ratio / balance• ‘quality’ assessed by the high-level criteria of Coherence and

Correspondence [not today]• Location of a JDM on the Task Structure dimension does

induce (?) (should induce?) matching process on Cognitive dimension

Highest demands and requirements

Lowest demands and requirements

Control,ability to

manipulatevariables

Timeand

resources(??)

Analyticalskillsand

capacities

Interventional-ObservationalBorder Conflicts

Qualitative-QuantitativeBorder Conflicts

MODELLING

ARGUMENTATION

INTUITION

INTERVENTION

OBSERVATION

INTERVENTION

‘INSTINCT’

‘Taking Into Account and Bearing In Mind’• ‘Taking things into account’• ‘Giving considerations due weight’• ‘Establishing the right balance’• ‘Keeping things in proportion’• ‘Taking a measured view’• ‘Bringing everything into the equation’• ‘Figuring it out’• ‘Seeking a degree of consensus’• ‘Gauging the impact’• ‘Making sure things add up’• ‘Summing up…’

• note how TIABIM DT, a basically qualitative discourse, is given a quantitative flavour

Key rule• The products of BTs in B-land (beliefs =knowledge,

evidence, judgments, opinions) and of PTs in P-land (preferences=values, importance weights) can only be input into a DT in D-land at the same latitude as that DT

• E.g. an effect size from an RCT created at the A-I ratio (level of explicitness and precision) of Randomistan can only become an input into TIABIMIA by being moved to the A-I ratio characteristic of Conferalot (Don’t be deceived by appearances or claims)

• E.g. an intuitive probability judgement from Judgia can only become an input into Analysia by being moved to the A-I ratio of Modelia

• Analysing up/down = Dumbing up/down ??

HEALTHIMPACTASSESSMENT

E-BGUIDELINES

Evidencebase

Patient’spreferences

Note the position of the apostrophe

HTAI

HTAI

TIABIM – the verdicts• TIABIM is fine, no problems, flaws• TIABIM is not perfect, it does produce lots of

bad/poor decisions, but this is because• the wrong people with the wrong values dominate –

bring in the true/right ‘stakeholders’ and it will be fine• we have the right people with the right values, but they

lack knowledge/information/evidence – supply them with better k/i/e and it will be fine

• we have the right people but TIABIM processes need improvement

• we have the right people but many currently lack the relevant TIABIM skills - build their capacity in these and it will be fine (classics education?)

BELIEFSYNTHESIS

PROBABILITIES

VALUESYNTHESIS

PREFERENCES?

HTAII

SCDA

HTAII

SCDA

TreatEff0.70

85.00; P = 0.13

TreatNotEff0.30

0.00; P = 0.05

TestPos0.90

X remains15.00; P = 0.02

TestNeg0.10

X0.20

TreatWorthless85.00; P = 0.08

TestPos0.10

NoTreatment100.00; P = 0.72

TestNeg0.90

NotX0.80

Test and Treat if Positive

89.81

TreatEff0.70

85.00

TreatNotEff0.30

0.00

X0.20

NotX0.80

85.00

Treat without testing

79.90

X0.20

15.00

NotX0.80

100.00

Neither Test nor Treat

83.00

Choice in possible case of XdisuTreat=1 5priorprobX=.2pTestNegg iv enNotX=.9PTestNegg iv enX=1-pTestPosgiven XpTestPosgiven No tX=1 -p TestNeg givenNotXpTestPosgiven X=.9pTreatEffective=.7uFH=100uX=1 5

Test and Treat if Positive : 89.81

Archetypal decis ion 2An otherwise fully healthy patient presents with symptoms and s igns which lead you to suspect either X or Z. Z is of no clinical s ignificance and the patient will fully recover very quickly with an hour's res t. We can therefore define Z as NotX.X has serious consequences if not treated quickly and successfully.A tes t exists for X but it is not perfect either in terms of sensitivity or specificity. We will assume it has no underiable aspects (disutility) as a process.A treatment exists for X but it is not certainly effective. Moreover it has some undesirable aspects (disutility) as a process. But if treated effectively the patient will return immediately to full health (FH)Evaluate the 3 options.

George Bernard Shaw The Doctor’s DilemmaSir Colenso Ridgeon has developed Opsonin, ‘works’ (?) but only when a test is done and result correctly interpretedLouis Dubedat : a dissolute artist and ‘bounder’ (needs opsonin)Blenkinsop : an outstanding GP who has dedicated his life to the poor … and so is poor (needs opsonin)BB : a very successful consultant who has dedicated his life to the rich and so is rich and doesn’t believe in testingJennifer Dubedat : the beautiful wife who Ridgeon falls in love with

George Bernard Shaw The Doctor’s DilemmaSir Colenso Ridgeon has developed Opsonin, ‘works’ (?) but ontest is done and result correctly interpretedLouis Dubedat : a dissolute artist and ‘bounder’ (needs opsoninBlenkinsop : an outstanding GP who has dedicated his life to thand so is poor (needs opsonin)BB : a very successful consultant who has dedicated his life to so is rich and doesn’t believe in testingJennifer Dubedat : the beautiful wife who Ridgeon falls in love w

The problem of the Single Criterion• Most common single criteria are Life

Expectancy/survival/mortality; ‘natural outcomes’; individual biomarkers; generalised ‘utility’…

• The QALY is a multi-attribute measure, bringing together Life Years and HRQOL (the Quality Adjustment)

• And the HRQOL measure is often based on a multi-dimensional instrument (e.g. Euroqol has 5)

• But it is still used as a single criterion

• Many other considerations are not ‘QALY-able’ –not incorporable into a ‘super QALY’

Patient-centred care• Requires patient’s preferences• If we use a population derived QALY tariff (set of weights

over the 5 dimensions of health) we are applying those of mean or median patient

• But it is not just that the QA is not for this patient… there are other considerations in their mind…• ‘Post-process’ Considerations (e.g. bother of follow-ups)• ‘In-Process’ Considerations (e.g. experience of surgery)• ‘Pre-Process’ Considerations (e.g. anxiety before surgery)

• Hazen’s GALY (Goal Achieved Life Years) is a recent attempt to add ‘extrinsic’ considerations to QALY (e.g. desire to live to see child’s wedding)

• Brazier, Dixon, Ratcliffe (2009) try to rescue SCDA• But it can’t deal with multiple considerations and certainly

not in a specific decision support role.

Note the position of the apostrophe!

Conclusion re SCDA• In context of either the clinical consultation OR an HTA the

necessary condition is that the stakeholder/s are able to indicate the relative importance of multiple considerations in real time with minimal complexity

• Requiring them to make numerous pair-wise comparisons or process alternative vignettes are simply not practical, despite the impressive normative arguments for them as rigorous elicitation methods

• Visual analog scales (or their equivalent) for separateattributes are the only serious contender

• SCDA is simply not a viable basis for a Clinical Decision Support System or a Health Technology Assessment

MCDA - variations• MCDA covers a multitude of different techniques• Mainly differ in the way they address issues like:

• the way criteria are assessed• the application and computation of weights• the algorithm used to derive the overall ranking• the model to describe individual preferences (compensatory

versus non-compensatory criteria, linear versus non-linear preferences)

• the uncertainty embedded in the data• the ability of stakeholders to participate in the process.

• Two emerge from my ‘MCDA’ quest in both contexts• Analytic Hierarchy Process• Annalisa

Baltussen and Niessen• The more analytical approaches that have been developed

over the past decades offer little guidance to policy makers.• Concentrate on single criteria such as effectiveness, cost-

effectiveness, burden/impact, equity• If address more than one criterion don’t advise on how to

integrate or judge the relative importance of each - policy makers need to make choices taking into account such multiple criteria simultaneously.

• Moreover, often do not cover all criteria that are relevant to policy makers (particularly ‘soft’ ones)

• In other disciplines MCDA (transport, agriculture, natural resource management) is routinely used in similar problems

TIABIMIA

ANALYSIA

van Til et al. 2008 p461

• “As was shown by Hummel et al. patients value different aspects of treatment compared with health professionals.”

[and compared with other patients] • “The AHP model used in this study could be a

way to include the personal aims, wishes and demands of each patient… different trade-offs regarding criteria importance could lead to a different preference for treatment.”

But…

• “The time requirements to complete the model [7 hours] were considered a disadvantage,

• and the AHP was thought to be bothersome for use in day-to-day decision making. This reservation might be a result of the large size of the current model. Simpler models might be possible, although care must be taken to maintain important details.”

Annalisa

Annalisa• An “intermediate” Decision Technology• Sits on boundary between Tiabimia and Analysia

- a ‘boundary object’ in itself• Makes possible optimal practical balancing

• intuition and analysis• rigour and relevance• complexity and practicality

• One screen picture of whole decision with interactivity

• Equally useful in HTA and CDM – important if link is ever to be established

Reprise…Step 1

TIABIMIA to ANALYSIAto increase analysis to intuition ratio

Step 2SCDA to MCDA

to increase coverage of attributes / outcomes / considerations not able to be synthesised into Single Criterion

Step 3AHP to Annalisa

to increase practicality, communicability…

Contexts• Clinical • Public• Public-clinical regulatory, e.g. NICE, PBAC, …• NICE currently struggling to deal with its responsibility

to ‘take into account’ multiple considerations in addition to clinical and cost-effectiveness

• Increasing emphasis on rigour of the Cost-effectiveness but with a QALY definition of effectiveness that leaves ‘other considerations’ to be processed with almost no rigour… e.g. terminality!

• Why not MCDA?

NICE attributes/considerations to be TIA• This exemplar set is derived from Guide to Technology Appraisal and other

NICE documents, including appeals (e.g. that on Bortezomib). Clearly it would be NICE’s task to come up with an appropriate set (as well as organize/outsource ratings and weightings) for an MCDA.• pClEff= probability that the NT is clinically effective relative to the

Comparator• pCostE20k= probability that the NT is cost effective relative to the

Comparator at a WTP below £20,000 per QALY• Acceptability/Appropriateness/Preferences [of patients and professionals]• Terminality= End of Life Use• Orph/NoAlt/Rescue= NT is 'orphan drug' OR has no alternative besides Best

Supportive Care OR is used in a 'Rule of Rescue' situation• OtherEq= Other Equity considerations• DHpriorities= clinical priority area as designated by Secretary of State for

Health and Welsh Assembly Government • HSFeasability/Impact= Health Service Feasibility/Impact• Innovatoriness• WiderSocietalConsiderations

DREA• HTA in the form of Cost Effective Analysis (CEA)

addresses the adoption decision: what is the best intervention to undertake, given this set of options, their Effectiveness, and a Willingness to Pay (WTP) Cost threshold per incremental unit of effectiveness?

• When undertaken stochastically and extended to VOIA (Value of Information Analysis) it simultaneously addresses the research decision: what is the best uncertain parameter to find out more about and how?

• In contrast, DREA addresses the decision decision - what is the best way to choose between intervention (and research) possibilities, given the alternative decision processes available and a decision resource threshold per unit of decision effectiveness?

• We can think of it as Value of Analysis Analysis

Practical usefulness(multi-attribute index based on key high-level criteria

of (a) social/individual fit and (b) time/resource constraints )

'Scientific'rigour

(multi-attribute indexbased on

key high-levelcriteria of

(a) logical coherence(b) empirical

correspondence)

0 1000

100 MinimumUsefulnessThreshold

Annalisa

AHP

CDA

CBDM

CJ

Choosing a D(S)T• The meta-decision faced by all practitioners and patients

and public policy makers ishow should we decide?

• There is no such thing as a ‘good decision process’ (or DT) in the abstract, only one that performs well on specified attributes with specified weights

• So MCDA can provide the framework within which to evaluate Decision Technologies (for a specified task in a specific context)

• DREA suggests that sometimes (often? usually?) Annalisa will be the optimal DT given the Willingness to Decision Resource

Why do economists ignore opportunity cost?• CEA/HTA1 (like most analytical techniques) continues to be

developed in pursuit of ever-greater theoretical/conceptual normativity and empirical/statistical rigour.

• It is taken for granted (e.g. Drummond, Schwartz, Jönsson et al 2008) that this continuous raising of the ‘best practice’ level by scientific standards is a good thing and it is occurring with remarkably little interest in, or attention to, either the ability and willingness of decision owners to resource best/good practice decision processes of this kind, or the complexity of decision owner’s maximand.

• Rex Brown has been one of the few people exploring the reasons for this growing gap and for the consequent failure of these advanced techniques to spread as widely as the professionals involved have urged and expected.

• His primary explanation lies in ‘analyst mismotivation / misalignment’, in the form of analysts’ pursuit of professional advancement/standing, mental comfort and financial well-being -which interact of course…

Descriptive

How close should ‘prescriptive’ be to ‘normative’ – and how far from

‘descriptive’?

Prescriptive?

Prescriptive?

Prescriptive?

Prescriptive?

Normative

Descriptive

Prescriptive?

Prescriptive?

Prescriptive?

Prescriptive?

Prescriptive?

Prescriptive?

Normative

WTDRX

WTDRY

WTDRZ

DE

DR

Predominantly Intuitive DT

Predominantly Analytical DT

rigour oughtis• an affliction in which the setting of unrealistically or

impractically rigorous requirements on some criterion or criteria leads to rejection of realistic / practical means of improvement … and hence maintenance of status quo

• often self-inflicted and cultivated as a defence of existing interests / competencies / processes

• but sometimes (as in many judgment and decision making situations) simply a reflection of misguided belief that 'scientific rigour' does not have to be incrementally traded-off with 'practical usefulness'.

•http://cafeannalisa.org.uk

References

• http://knol.google.com for my knols(key ‘dowie’ into search box)

Brown (2006) ‘Making decision research useful – not just rewarding’ Judgment and Decision Research 1 (2) 162-173

Brazier, Dixon and Ratcliffe (2009) ‘The role of patient preferences in Cost-Effectiveness Analysis’ Pharmacoeconomics 27 (9) 705-712

Drummond, Schwartz, Jönsson et al (2008) ‘Key principles for the improved conduct of Health Technology Assessments for resource allocation decisions’ International Journal of Technology Assessment in Health Care 24 (3) 244-258

Hazen (2007) ‘Adding Extrinsic Goals to the Quality-Adjusted Life Year Model’ Decision Analysis 4 (1) 3-16