Economic evaluation. Methods for adjusting survival estimates in the presence of treatment...
-
Upload
htai-bilbao-2012 -
Category
Health & Medicine
-
view
766 -
download
0
description
Transcript of Economic evaluation. Methods for adjusting survival estimates in the presence of treatment...
Methods for adjusting survival estimates in the presence of treatment crossover: a simulation studyNicholas Latimer, University of SheffieldCollaborators: Paul Lambert, Keith Abrams, Michael Crowther, Allan Wailoo and James Morden
Contents
What is the treatment crossover problemPotential solutionsSimulation studyResultsConclusions
Treatment switching – the problem
In RCTs often patients are allowed to switch from the control treatment to the new intervention after a certain timepoint (eg disease progression)
OS estimates will be confounded
OS is a key input into the QALY calculation
Cost effectiveness results will be inaccurate an ITT analysis is likely to underestimate the treatment benefit
Inconsistent and inappropriate treatment recommendations could be made
Treatment switching – the problem
Survival time
Control Treatment
Intervention
Control Intervention
PFS
PFS
PFS
PPS
PPS
PPS
True OS difference
RCT OS differenc
e
Switching is likely to result in an underestimate of the treatment effect
What is usually done to adjust?No clear consensus
Numerous ‘naive’ approaches have been taken in NICE appraisals, eg:
Take no action at allExclude or censor all patients who crossover
Occasionally more complex statistical methods have been used, eg:Rank Preserving Structural Failure Time Models (RPSFTM)Inverse Probability of Censoring Weights (IPCW)
And others are available from the literature, eg:Structural Nested Models (SNM)
Very prone to selection bias – crossover isn’t random
What are the consequences?
TA 215, Pazopanib for RCC [51% of control switched]
ITT: OS HR (vs IFN) = 1.26 ICER = Dominated
Censor patients: HR = 0.80 ICER = £71,648
Exclude patients: HR = 0.48 ICER = £26,293
IPCW: HR = 0.80 ICER = £72,274
RPSFTM: HR = 0.63 ICER = £38,925
Potential solutions
RPSFTM (and IPE algorithm)Randomisation-based approach, estimates counterfactual survival times
Key assumption: common treatment effect
IPCWObservational-based approach, censors xo patients, weights remaining patients
Key assumptions: “no unmeasured confounders”; must model OS and crossover
SNMObservational version of RPSFTM
Key assumptions: “no unmeasured confounders”; must model OS and crossover
Simulation study (1)None of these methods are perfectBut we need to know which are likely to produce least bias in different scenarios
Simulation study
Simulate survival data for two treatment groups, applying crossover that is linked to patient characteristics/prognosisIn some scenarios simulate a treatment effect that changes over timeIn some scenarios simulate a treatment effect that remains constant over timeTest different %s of crossover, and different treatment effect sizes
How does the bias and coverage associated with each method compare?
Simulation study (2)Methods assessed
Naive methodsITTExclude crossover patientsCensor crossover patientsTreatment as a time-dependent covariate
Complex methodsRPSFTMIPE algorithmIPCWSNM
Results: common treatment effectRPSFTM / IPE worked very well
IPCW and SNM performed ok when crossover % was lower
IPCW and SNM performed poorly when crossover % was very high
Naive methods performed poorly (generally led to higher bias than ITT)
-10.00
0.00
10.00
20.00
30.00
40.00
50.00
3 4 7 8 15 16 19 20 27 28 31 32 39 40 43 44 51 52 55 56 63 64 67 68
AUC
mea
n bi
as (%
)
ITT IPCW RPSFTM IPE SNM
Results: effect 15% in xo patientsRPSFTM / IPE produced higher bias than previous scenarios
IPCW and SNM performed similarly to RPSFTM / IPE providing crossover < 90%
IPCW and SNM performed poorly when crossover % was very high
Bias not always lower than that associated with the ITT analysis
-15.00
-5.00
5.00
15.00
25.00
35.00
45.00
1 2 5 6 13 14 17 18 25 26 29 30 37 38 41 42 49 50 53 54 61 62 65 66
AUC
mea
n bi
as (%
)
ITT IPCW RPSFTM IPE SNM
RPSFTM / IPE produced even more bias
IPCW and SNM produce less bias than RPSFTM / IPE providing crossover < 90%
No ‘good’ options when crossover % is very high
Often ITT analysis likely to result in least bias (esp. when trt effect low)
-20.00
-10.00
0.00
10.00
20.00
30.00
40.00
50.00
9 10 11 12 21 22 23 24 33 34 35 36 45 46 47 48 57 58 59 60 69 70 71 72
AUC
mea
n bi
as (%
)
ITT IPCW RPSFTM IPE SNM
Results: effect 25% in xo patients
ConclusionsTreatment crossover is an important issue that has come to the fore in HE arena
Current methods for dealing with treatment crossover are imperfect
Our study offers evidence on bias in different scenarios (subject to limitations)
RPSFTM / IPE produce low bias when treatment effect is common But are very sensitive to this
IPCW / SNM are not affected by changes in treatment effect between groups, but in (relatively) small trial datasets observational methods are volatile Especially when crossover % is very high (leaving low n in control group)
Very important to assess trial data, crossover mechanism, treatment effect to determine which method likely to be most appropriate
Don’t just pick one!!
Back-up Slides
Data Generation (1)Used a two-stage Weibull model to generate underlying survival times and a time-dependent covariate (called ‘CEA’)Longitudinal model for CEA (for ith patient at time t):
where is the random intercept is the slope for a patient in the control arm is the slope for a patient in the treatment arm (all is the change in the intercept for a patient with bad prognosis
compared to a patient without bad prognosis Picked parameter values such that CEA increased over time, more slowly in the experimental group, and was higher in the badprog group
Data Generation (2)The survival hazard function was based upon a Weibull (see Bender et al 2005):
In our case,
where is the log hazard ratio (the baseline treatment effect) is the time-dependent change in the treatment effect
is the impact of a bad prognosis baseline covariate on survival
is the coefficient of CEA, indicating its effect on survivalWe used this to generate our survival times
So, CEA has an effect on survival over time, and there is a separate time-dependent treatment effect
) (cea(t)+badprog+trt*) log(t)*(+trt*= i2ii1 X1
2
)exp()( 1 Xtth
Data Generation (3)We applied a reduced treatment effect to crossover patients – this was equivalent to the baseline treatment effect multiplied by a factor to ensure that this effect was less than (or equal to) the average effect received in the experimental group
Data Generation (4)
We then selected parameter values in order that ‘realistic’ datasets were created:
0.00
0.20
0.40
0.60
0.80
1.00
Sur
viva
l pro
babi
lity
249 186 126 71 42 21 0trtrand = 1251 154 86 52 25 9 0trtrand = 0
Number at risk
0 200 400 600 800 1000 1200analysis time
trtrand = 0 trtrand = 1
0
0.5
1
1.5
2
2.5
0 200 400 600 800 1000 1200
Acc
eler
ation
Fac
tor
Time (days)
AF actual
AF predicted
Data Generation (5)We made several assumptions about the ‘crossover mechanism’:
1. Crossover could only occur after disease progression (disease progression was approximately half of OS, calculated for each patient using a beta(5,5) distribution)
2. Crossover could only occur at 3 ‘consultations’ following disease progression
These were set at 21 day intervalsProbability of crossover highest at initial consultation, then falls in second and third
3. Crossover probability depended on time-dependent covariates:CEA value at progression (high value reduced chance of crossover)Time to disease progression (high value increased chance of crossover)This was altered in scenarios to test a simpler mechanism where probability only depended on CEA
Given all this, CEA was a time-dependent confounder
ScenariosVariable Value AlternativeSample size 500 Number of prognosis groups (prog) 2 Probability of good prognosis 0.5 Probability of poor prognosis 0.5 Maximum follow-up time 3 years (1095 days) Multiplication of OS survival time due to bad prognosis group
Log hazard ratio = 0.5
Survival time distribution Alter parameters to test two levels of disease severity
Initially assumed treatment effect Alter to test two levels of treatment effect Time-dependence of treatment effect Treatment effect received is equal in all crossover
patients, and equals baseline treatment effect multplied by a factor. However set α to zero in some scenarios. Also include additional treatment effect decrement in crossover patients in some scenarios
Probability of switching treatment over time
Test two levels of treatment crossover proportions
Prognosis of crossover patients Test three crossover mechanisms in which different groups become more likely to cross over
ScenariosVariable Value AlternativeSample size 500 Number of prognosis groups (prog) 2 Probability of good prognosis 0.5 Probability of poor prognosis 0.5 Maximum follow-up time 3 years (1095 days) Multiplication of OS survival time due to bad prognosis group
Log hazard ratio = 0.5
Survival time distribution Alter parameters to test two levels of disease severity
(2)
Initially assumed treatment effect Alter to test two levels of treatment effect Time-dependence of treatment effect Treatment effect received is equal in all crossover
patients, and equals baseline treatment effect multplied by a factor. However set α to zero in some scenarios. Also include additional treatment effect decrement in crossover patients in some scenarios
Probability of switching treatment over time
Test two levels of treatment crossover proportions
Prognosis of crossover patients Test three crossover mechanisms in which different groups become more likely to cross over
ScenariosVariable Value AlternativeSample size 500 Number of prognosis groups (prog) 2 Probability of good prognosis 0.5 Probability of poor prognosis 0.5 Maximum follow-up time 3 years (1095 days) Multiplication of OS survival time due to bad prognosis group
Log hazard ratio = 0.5
Survival time distribution Alter parameters to test two levels of disease severity
(2)
Initially assumed treatment effect Alter to test two levels of treatment effect (4)Time-dependence of treatment effect Treatment effect received is equal in all crossover
patients, and equals baseline treatment effect multplied by a factor. However set α to zero in some scenarios. Also include additional treatment effect decrement in crossover patients in some scenarios
Probability of switching treatment over time
Test two levels of treatment crossover proportions
Prognosis of crossover patients Test three crossover mechanisms in which different groups become more likely to cross over
ScenariosVariable Value AlternativeSample size 500 Number of prognosis groups (prog) 2 Probability of good prognosis 0.5 Probability of poor prognosis 0.5 Maximum follow-up time 3 years (1095 days) Multiplication of OS survival time due to bad prognosis group
Log hazard ratio = 0.5
Survival time distribution Alter parameters to test two levels of disease severity
(2)
Initially assumed treatment effect Alter to test two levels of treatment effect (4)Time-dependence of treatment effect Treatment effect received is equal in all crossover
patients, and equals baseline treatment effect multplied by a factor. However set α to zero in some scenarios. Also include additional treatment effect decrement in crossover patients in some scenarios
(8)
(12)
Probability of switching treatment over time
Test two levels of treatment crossover proportions
Prognosis of crossover patients Test three crossover mechanisms in which different groups become more likely to cross over
ScenariosVariable Value AlternativeSample size 500 Number of prognosis groups (prog) 2 Probability of good prognosis 0.5 Probability of poor prognosis 0.5 Maximum follow-up time 3 years (1095 days) Multiplication of OS survival time due to bad prognosis group
Log hazard ratio = 0.5
Survival time distribution Alter parameters to test two levels of disease severity
(2)
Initially assumed treatment effect Alter to test two levels of treatment effect (4)Time-dependence of treatment effect Treatment effect received is equal in all crossover
patients, and equals baseline treatment effect multplied by a factor. However set α to zero in some scenarios. Also include additional treatment effect decrement in crossover patients in some scenarios
(8)
(12)
Probability of switching treatment over time
Test two levels of treatment crossover proportions (24)
Prognosis of crossover patients Test three crossover mechanisms in which different groups become more likely to cross over
ScenariosVariable Value AlternativeSample size 500 Number of prognosis groups (prog) 2 Probability of good prognosis 0.5 Probability of poor prognosis 0.5 Maximum follow-up time 3 years (1095 days) Multiplication of OS survival time due to bad prognosis group
Log hazard ratio = 0.5
Survival time distribution Alter parameters to test two levels of disease severity
(2)
Initially assumed treatment effect Alter to test two levels of treatment effect (4)Time-dependence of treatment effect Treatment effect received is equal in all crossover
patients, and equals baseline treatment effect multplied by a factor. However set α to zero in some scenarios. Also include additional treatment effect decrement in crossover patients in some scenarios
(8)
(12)
Probability of switching treatment over time
Test two levels of treatment crossover proportions (24)
Prognosis of crossover patients Test three crossover mechanisms in which different groups become more likely to cross over
(72)
This combined to 72 scenarios
ScenariosVariable Value AlternativeSample size 500 Number of prognosis groups (prog) 2 Probability of good prognosis 0.5 Probability of poor prognosis 0.5 Maximum follow-up time 3 years (1095 days) Multiplication of OS survival time due to bad prognosis group
Log hazard ratio = 0.5
Survival time distribution Alter parameters to test two levels of disease severity
Initially assumed treatment effect Alter to test two levels of treatment effect Time-dependence of treatment effect Treatment effect received is equal in all crossover
patients, and equals baseline treatment effect multplied by a factor. However set α to zero in some scenarios. Also include additional treatment effect decrement in crossover patients in some scenarios
Probability of switching treatment over time
Test two levels of treatment crossover proportions
Prognosis of crossover patients Test three crossover mechanisms in which different groups become more likely to cross over
Estimating AUC
1. ‘Survivor function’ approachApply treatment effect to survivor function (or hazard function) estimated for experimental group calculate AUC
2. ‘Extrapolation’ approachExtrapolate counterfactual dataset to required time-point (only relevant for RPSFTM/IPE approaches) calculate AUC
3. ‘Shrinkage’ approachUse estimated acceleration factor to ‘shrink’ survival times in crossover patients in order to obtain an adjusted dataset calculate AUC (only relevent for AF-based approaches)
28A topical analogy...
VsEngland
(control group)Spain
(intervention group)
Kick-off...
29A topical analogy...
Halftime: England 0 – 3 Spain
Spain are statistically significantly better than England For ethical reasons, the Spanish team are cloned and the English team are sent home. So the second half is Spain Vs Spain
30A topical analogy...
Switching is likely to result in an underestimate of the true supremacy of the intervention
Full time: England 2 – 5 Spain
Spain still win, but not as comfortably as they would have
09/04/2023© The University of Sheffield
31
Results (4)While the SNM and IPCW methods appeared to produce similar levels of bias, the SNM approach was more volatile:
Relatively often failed to converge (in up to 90% of sims in one scenario)Typically when disease severity was high and treatment effect was low
Only produced lower bias than the ITT analysis in 38% of scenarios (compared to 60% for the IPCW method)
Was more sensitive to increasing crossover %
09/04/2023© The University of Sheffield
32
Results (5)Relationship between bias and treatment crossover %
-40
-20
0
20
40
60
80
100
120
140
60% 70% 80% 90% 100%
Mea
n %
bia
s
Crossover proportion (at-risk patients)
ITT
09/04/2023© The University of Sheffield
33
Results (5)Relationship between bias and treatment crossover %
-40
-20
0
20
40
60
80
100
120
140
60% 70% 80% 90% 100%
Mea
n %
bia
s
Crossover proportion (at-risk patients)
IPE
RPSFTM
ITT
09/04/2023© The University of Sheffield
34
Results (5)Relationship between bias and treatment crossover %
-40
-20
0
20
40
60
80
100
120
140
60% 70% 80% 90% 100%
Mea
n %
bia
s
Crossover proportion (at-risk patients)
IPCW
IPE
RPSFTM
ITT
09/04/2023© The University of Sheffield
35
Results (5)Relationship between bias and treatment crossover %
-40
-20
0
20
40
60
80
100
120
140
60% 70% 80% 90% 100%
Mea
n %
bia
s
Crossover proportion (at-risk patients)
IPCW
IPE
RPSFTM
SNM
ITT
36ConclusionsTreatment crossover is an important issue that has come to the fore in HE arena
Current methods for dealing with treatment crossover are imperfect
Our study offers evidence on bias in different scenarios (subject to limitations)
1. RPSFTM / IPE produce low bias when treatment effect is common
2. When treatment effect ≈ 15% lower in crossover patients RPSFTM / IPE and IPCW / SNM methods produce similar levels of bias (5-10%) [provided suitable data available for obs methods and <90% crossover]
3. When treatment effect ≈ 25% lower IPCW/SNM produce less bias than RPSFTM/IPE[provided suitable data available for obs methods and <90% crossover][and significant bias likely to remain ITT analysis may offer least bias]
4. Very important to assess trial data, crossover mechanism, treatment effect to determine which method likely to be most appropriate
Don’t just pick one!!