Should Suboptimal Clinical

Evidence be Used to Inform

HTA Recommendations?

Alex Chambers

Director, pCODR

CADTH

Disclosure

• The speaker has no financial or other conflict of interest to

report.

1

Overview of Drug Review in Canada

Health Canada

Regulator (Effect & safety)

CADTH (CDR and pCODR)

INESSS (Quebec)

HTA (Assess value)

F/P/T Ministries of Health and Provincial Cancer Agencies

Decision maker/ funder

Pan Canadian Pharmaceutical Alliance (pCPA)

Price negotiator

2

Does pCODR Review

Submissions with Suboptimal

Clinical Evidence?

4

Source: Tiruneh et al, ASCO Quality, 2018

pERC Recommendations Based on

Non-Randomized Trials (Hematology-only)

Cost of Drugs Without a Direct Comparison

to Standard of Care

5

$-

$5,000

$10,000

$15,000

$20,000

$25,000

$30,000

$35,000

$40,000

List

Pri

ce p

er 2

8 d

ays

2012 2015 2016 2017 2013 2014

Guidance from 2016 pCODR Metrics

Report

• If non-comparative data are used, it is important for a

submitter to consider the following factors:

• If there is an important therapeutic need that a drug

under review may fill

• A randomized trial is not feasible due to various reasons

(e.g., the rarity of disease, unethical to randomize,

among others)

• Meaningful outcomes in a large sample of patients

when compared with historical data on current standard

of care.

6

What are the Challenges of

Reviewing Suboptimal Clinical

Evidence?

pCODR Expert Review Committee’s

(pERC) Deliberative Framework

8

Clinical Benefit • Effectiveness • Safety • Burden of Illness • Need

Patient Values • Alignment with values

Economic Evaluation • Cost effectiveness

Adoption and Feasibility • Economic feasibility (BIA) • Organizational feasibility

Example 1: Avelumab for Merkel Cell

Carcinoma

9

pERC Recommendation Avelumab

cont’d… • pERC concluded that there is a substantial unmet need

for alternative options

• Conducting a RCT in this setting with avelumab compared

with palliative chemotherapy would not be feasible given

the small population of patients with mMCC.

• pERC noted that a substantial proportion of patients

experienced an objective response rate (ORR), which is an

important and clinically meaningful outcome in mMCC.

pERC noted that the ORR was higher than historical

responses observed with chemotherapy treatments used

to treat mMCC.

10

pERC’s Next Step for Avelumab

• Collecting Evidence to Reduce Uncertainty in the

Magnitude of Clinical Benefit and the Cost-

Effectiveness of Avelumab

• Additional prospective evidence should be collected to

decrease the uncertainty in the incremental effect and

provide a greater understanding of the true cost-

effectiveness of avelumab. Specific information on long-

term efficacy, safety, and QoL would be of particular value.

11

Example 2: Nivolumab for

Hepatocellular Carcinoma

12

pERC Recommendation on

Nivolumab cont’d…

• pERC concluded that there is a significant unmet need in

this setting

• pERC discussed the feasibility of conducting a phase III

RCT in this patient population. They considered that there

are RCTs investigating other therapies for this patient

population. pERC also considered that registered clinicians

noted that they wanted access to nivolumab, but recognized

the uncertainty of the non-comparative data.

• [pERC] noted that the ORR … did not reach the clinical

significance threshold

13

Conclusions

• pCODR reviews all of the available evidence for drugs

under review

• pERC regularly makes positive, conditional, and negative

recommendations on non-comparative data

• Moving forward…

• Opportunities to re-review the evidence, especially in

circumstances were there is lower quality of evidence,

would be worthwhile to ensure that drugs are delivering

the value to the health system that was expected

14

15

© 2018 Evidera. All Rights Reserved.

Incorporating evidence quality into pCODR decisions using an MCDA

Kevin Marsh Executive Director, Patent Centered Research 2019 CADTH Symposium

Objectives

Premise: Society is willing to provide early access, based on sub-optimal evidence for patients with unmet need

Objective:

1. Is there any evidence that this is the case?

2. How might we more formally operationalize such preference in decision making?

What factors influence pCODR decisions?

18

What factors influence pCODR decisions? Methods: Data extraction

19

Identify variables (56)

Test extraction Extract (22

items, decisions 7/10-4/18)

Analysis

Data were extracted on the variables focusing on four main areas:

Clinical Benefit Economic Evaluation

Patient Inputs Adoption Feasibility

Quality of clinical evidence ● Number of trials ● Trial design

Clinical outcomes ● PFS/OS/RR ● Safety ● QoL

Quality of economic evaluation ● Submission quality ● Limitations

Economic evaluation result ● Manufacturer- submitted

ICER ● HTA re-analyzed ICER ● Conclusion of CE

Alignment with patients’ value in terms of

● Disease control / symptom relief

● Drug admin type ● Access to new treatment ● Side effect ● Patient QoL

Quality of budget impact analysis

Magnitude of budget impact

Abbreviations: admin = administration; HTA = health technology assessment; ICER = incremental cost-effectiveness ratio; OS = overall survival; pCODR = pan-Canadian Oncology Drug Review; PFS = progression-free survival; QoL = quality of life; RR = response rate

What factors influence pCODR decisions? Statistical Analyses: Overview

Of the 111 recommendation decisions published, 12 were full list, 74 conditional list, and 25 do not list.

Our predictive analyses assumed that the pCODR decisions were conducted in a sequential manner:

Two methodologies were explored to model each stage of the decision process:

● Logit binary regression model using Cordeiro and McCullagh bias correction1

● Classification and Regression Trees (CART)

Decision

Conditional List Fully List

Rejection Approval

Source: 1. Cordeiro, G. M. and P. McCullagh (1991). Bias correction in generalized linear models. Journal of the Royal Statistical Society, Series B: Methodological 53(3), 629–643.

20

Results: Predictive Model - List vs. Do Not List

21

Results: Predictive Model - Full List vs. Conditional List

22

Can MCDA support pCODR to make decisions?

23

What is MCDA?

24

A collection of analytical methods used to support decision

making and value communication in the context of:

• Complex problems involving multiple, potentially conflicting,

objectives

• Involving potentially multiple stakeholders

What is MCDA? Common Steps

25

Problem structuring

Measure treatment

performance

Elicit stakeholder preferences

Evaluate and compare

alternatives

Overall Objective

Benefit Risk Convenience

Treatment A

Treatment C

Treatment B

𝑤𝑏 𝑤𝑐 𝑤𝑟

• Identify treatments

• Stakeholders

• Criteria

• Performance

Overall performance

score

Treatment A

Treatment C

Treatment B

Analysis contributes here

Benefits of the MCDA Process

26

Ensures all relevant factors (including intangibles) are

considered

A transparent link between performance, judgments and value

Quantify priorities and preferences Fosters a shared understanding of the decision problem and identifies

areas of important disagreement

Better decisions (consistent, transparent)

How is MCDA used to support HTA? There is precedent for the adoption of MCDA

27

IQWiG: 2 types of MCDA “can contribute to determining the most important outcomes for patients as part of economic evaluation”

Colombia: Developing an MCDA for HTA

Thailand: MCDA used to inform coverage decisions for HIV/AIDS interventions

Russia: MCDA for orphan drugs

* Illustrative examples; not comprehensive, IQWiG: The Institute for Quality and Efficiency in Healthcare

Hungary: MCDA has been used to evaluate new hospital medical technologies since 2010

Indonesia

Ghana

Cote d’Ivoire

South Africa

Italy: The Lombardy region introduced MCDA in 2008 to decide on the introduction and delisting of health technologies

New Zealand: MCDA approach to ‘points system’ for elective surgery

NHS England MCDA used in commissioning specialised services

Implemented in 2010

Evaluation of 14 applications for medical technology

Criteria and weights established by a committee comprising of:

● The Health Financing Agency

● The Ministry of Health

● Clinical experts

● Health economists

How is MCDA used to support HTA? Case Study: Reimbursing Medical Technology in Hungary

Source: Endrei, et al. Value Health 2014

28

Criteria Levels Weight SOC Treatment

High quality evidence? Clinical benefit

Yes, Improved 4

No, Improved 3

No improvement 0

Side effects Aligned 2

Not aligned 0

QoL

Improved 2

Similar 1

Deteriorated 0

Unmet need Yes 2

No 0

8 9

$X $Y

29

What might an MCDA look like?

ICER: (Y-X)/(9-8)

How might decision makers respond to the MCDA proposal? Consultations have identified some challenges

30

1. Consultation on the Development of 2013 Guide to the Methods of Technology Appraisal.

o Benefits of MCDA (consistency) outweighed by ‘advantages of deliberation’ flexibility to undertake context specific decisions

2. Consultation on the Implementation of Value Based Assessment (1)

o Allow conditions with a higher burden and societal impact to obtain a higher threshold of up to £50,000/QALY?

(1) https://www.ispor.org/ValueOutcomesSpotlightResources/health-policy_value-based-assessment-UK.pdf

“too formulaic with not enough flexibility”

“lack of standard approach to elicit weights”

“difficult to find weights that satisfied all members of society”

31

The pERC Deliberative Framework [9] identifies four main criteria that need to be considered during pERC’s review process

Clinical Benefit Economic Evaluation

Patient Inputs Adoption Feasibility

Quality of clinical evidence ● Number of trials

submitted/considered by pCODR

● Trial design Clinical outcomes

● PFS/OS/RR ● Safety ● QoL

Quality of economic evaluation ● Submission quality ● Limitations

Economic evaluation result ● Manufacturer- submitted

ICER ● HTA re-analyzed ICER ● Conclusion of cost-

effectiveness

Alignment with patients’ value in terms of

● Disease control / symptom relief

● Drug admin type ● Access to new treatment ● Side effect ● Patient QoL

Quality of budget impact analysis

Magnitude of budget impact

Abbreviations: admin = administration; HTA = health technology assessment; ICER = incremental cost-effectiveness ratio; OS = overall survival; pCODR = pan-Canadian Oncology Drug Review; PFS = progression-free survival; QoL = quality of life; RR = response rate

How might decision makers respond to the MCDA proposal? A complete framework?

How might decision makers respond to the MCDA proposal? Are attribute levels sufficiently nuanced?

32

Value

Clinical outcomes/trial quality

Clinical benefit by trial quality

Clinical efficacy

(clinical benefit vs. no benefit)

PFS

OS

RR

Trial quality (high vs. low)

Phase

Phase III

Phase II or I

Randomization

Randomized

Non-randomized

Control arm

Active/placebo controlled

Single arm

QoL

Deteriorated/ uncertain/

limited

Improved

Similar/ maintain/stable

Patient value-side effect

No (not aligned)

Unknown

Yes (aligned)

Unmet need

Yes

No

Criteria Levels

Trial Quality

High – RCT, blinded

Low

Are HTA value frameworks additive? E.g. The value of health gain is a function of baseline severity?

33

Nord E, Daniels N, Kamlet M. QALYs: some challenges. Value in Health 2009; 12: S10–5.

Rowen et al http://journals.sagepub.com/doi/full/10.1177/0272989X15619389

Are HTA value frameworks additive? Additive frameworks can produce erroneous results

34

Overall Objective

Health gain

Severity # Patients

40% × 40 + 40% × 70 + 20% × 40 = 52

𝑤𝑏 = 40% 𝑤𝑝=20% 𝑤𝑠 = 40%

100

0

40

70

40

Are HTA value frameworks additive? Additive frameworks can produce erroneous results

35

Overall Objective

Health gain

Severity # Patients

40% × 40 + 40% × 70 + 20% × 40 = 52

𝑤𝑏 = 40% 𝑤𝑝=20% 𝑤𝑠 = 40%

100

0

40

70

40

40% × 0 + 40% × 100 + 20% × 100 = 60!

100 100

E.g. Morton A (2017) Treacle and Smallpox: Two Tests for Multicriteria Decision Analysis Models in Health Technology Assessment. Value in Health 512 – 515.

How might decision makers respond to the MCDA proposal? Value concepts are not additive?

36

Value

Clinical outcomes/trial quality

Clinical benefit

Trial quality (high vs. low)

QoL

Patient value-side effect

Unmet need

Value interactions/ overlaps

A proposal ‘MCDA with (multiplicative) decision rules’

37

Unmet need Trial quality ICER Threshold

High High A

Low B

Medium High C

Low D

Low High E

Low F

Step 1: ‘Rules’

Step 2: Deliberation

Factors for which there is some consensus on their relative importance

A. Validate / refine framework with pCODR

B. Stated preference

List of other factors considered important, but for which it is not possible to agree weights

Should MCDA be used to support HTA decision making?

Christopher McCabe PhD

April 15 2019

Conflicts of Interest

The Institute of Health Economics is a Not-for-Profit founded by the Government of Alberta, the Universities of Alberta and Calgary, and the Canadian Pharmaceutical Industry. It receives core funding from the Government of Alberta, the Universities of Alberta and Calgary and Canadian pharmaceutical companies who are members of the Institute

In addition, the Institute receives project specific funding from public and private sector organizations.

Dr. McCabe receives research funding from CIHR, Genome Canada, Alberta Innovates and the UK National Institute for Health Research

The views expressed in this presentation do not necessarily represent the views of our Board Member Organizations or funders

Overview

• Multi-factorial decision making

– The challenge of preference independence

– Appropriate use of evidence

• Factor weights in MCDA

• Values and MCDA for Health Technology Appraisal

Multi-factorial Decision Making

Mutual Independence of Preferences: 1. The value attached to side effects are

unlikely to be independent of the value attached to changes in quality of life.

2. The value attached to any given clinical benefit is unlikely to be independent of the value attached to addressing unmet need.

3. The value attached to a given clinical benefit is unlikely to be independent of the value attached to a given change in quality of life – although the value attached to an improvement in quality of life may well be independent of the value attached to the clinical benefit that produced it.

Multi-factorial Decision Making

Inte

rnal

val

idit

y External validity

Trial Quality

Multi-factorial Decision Making

Factor weights in MCDA

Life threatening acute disease gets twice the weight of non-life threatening chronic disease.

The total budget impact is only as important as the reputational impact.

Cost Effectiveness is twice as important as any other health policy priority

Are these values the right values?

Values and health care resource allocation

Values and health care resource allocation

1. What is to be valued?

2. How is it to be described?

3. How is it to be valued

4. Who is to value it?

5. How are values for all possible combinations to be generated?

6. How are the valuations to be aggregated?

MCDA in HTA: some challenges

• Should weights represent values/preferences? – If not, what should they represent and why?

• Does the wider scope of MCDA obviate the need to choose: – How values are elicited – Whose values are used for the decision – Methods for combining and extrapolating from sample value data

• Does the wider scope of MCDA capture opportunity cost any more convincingly than standard HTA?

• Are MCDA models any more accurate at predicting actual preferences?

THANK YOU