Dr Caroline Sabin, Royal Free Hospital UK-CAB - 13 August 2004- Session 5: The design of RCTs of...
-
date post
19-Dec-2015 -
Category
Documents
-
view
218 -
download
1
Transcript of Dr Caroline Sabin, Royal Free Hospital UK-CAB - 13 August 2004- Session 5: The design of RCTs of...
Dr Caroline Sabin, Royal Free HospitalDr Caroline Sabin, Royal Free Hospital UK-CAB - 13 August 2004- www.i-Base.infoUK-CAB - 13 August 2004- www.i-Base.info
Session 5: The design of RCTs of Session 5: The design of RCTs of treatments for HIV infectiontreatments for HIV infection
Caroline SabinCaroline Sabin
Reader in Medical Statistics and EpidemiologyReader in Medical Statistics and EpidemiologyDepartment of Primary Care and Population Sciences, Department of Primary Care and Population Sciences,
RF&UCMS RF&UCMS
A beginners guide to some of the A beginners guide to some of the methodological and statistical methodological and statistical
issues in HIV researchissues in HIV research
Dr Caroline Sabin, Royal Free HospitalDr Caroline Sabin, Royal Free Hospital UK-CAB - 13 August 2004- www.i-Base.infoUK-CAB - 13 August 2004- www.i-Base.info
What is a clinical trial?What is a clinical trial?
Any form of Any form of planned experimentplanned experiment which involves which involves patientspatients and is designed to find the and is designed to find the most appropriate most appropriate
treatmenttreatment for a particular medical condition for a particular medical condition
Dr Caroline Sabin, Royal Free HospitalDr Caroline Sabin, Royal Free Hospital UK-CAB - 13 August 2004- www.i-Base.infoUK-CAB - 13 August 2004- www.i-Base.info
Types of trials (clinical)Types of trials (clinical)
Phase I studies Focus on safety rather than efficacyDose-escalation studies, studies of drug metabolism and bioavailabilityUsually based on small numbers of subjects, often healthy volunteers
Phase II studiesInitial investigation for clinical effect. Small-scale studies into effectiveness and safety of drug.
Phase III studiesFull-scale treatment evaluation. Comparison to standard therapy (if one exists) or placebo
Phase IV trials Post-marketing surveillance. Monitoring for adverse effects. Long-term studies of morbidity and mortality. Promotion exercises
Dr Caroline Sabin, Royal Free HospitalDr Caroline Sabin, Royal Free Hospital UK-CAB - 13 August 2004- www.i-Base.infoUK-CAB - 13 August 2004- www.i-Base.info
Topics already covered in first sessionTopics already covered in first session
Control groupsControl groups
RandomisationRandomisation
BlindingBlinding
Parallel vs. cross-over trialsParallel vs. cross-over trials
The limitations of RCTsThe limitations of RCTs
Dr Caroline Sabin, Royal Free HospitalDr Caroline Sabin, Royal Free Hospital UK-CAB - 13 August 2004- www.i-Base.infoUK-CAB - 13 August 2004- www.i-Base.info
Topics to be covered todayTopics to be covered today
Why do we need a control group?Why do we need a control group?
Why do we need randomisation?Why do we need randomisation?
The protocolThe protocol
Defining endpoints (primary and secondary Defining endpoints (primary and secondary endpoints, clinical vs surrogate endpoints)endpoints, clinical vs surrogate endpoints)
How to deal with ‘protocol violations’ (patients who How to deal with ‘protocol violations’ (patients who drop out of the study and missing data)drop out of the study and missing data)
Approaches to analysis (ITT, as treated)Approaches to analysis (ITT, as treated)
Subgroup and interim analysesSubgroup and interim analyses
Dr Caroline Sabin, Royal Free HospitalDr Caroline Sabin, Royal Free Hospital UK-CAB - 13 August 2004- www.i-Base.infoUK-CAB - 13 August 2004- www.i-Base.info
Why do we need a control groupWhy do we need a control group
Early medical developments were usually so Early medical developments were usually so great that controls weren’t always needed (eg. great that controls weren’t always needed (eg. trials of anaesthetics, first trials of antibiotics trials of anaesthetics, first trials of antibiotics etc.)etc.)
However, most developments these days are However, most developments these days are more modest and some form of control group is more modest and some form of control group is now essentialnow essential
Dr Caroline Sabin, Royal Free HospitalDr Caroline Sabin, Royal Free Hospital UK-CAB - 13 August 2004- www.i-Base.infoUK-CAB - 13 August 2004- www.i-Base.info
Why do we need a control groupWhy do we need a control group
Silverman, 1985 – Epidemic of retrolental fibroplasia Silverman, 1985 – Epidemic of retrolental fibroplasia in babiesin babies
Uncontrolled trials suggested that treatment with Uncontrolled trials suggested that treatment with adrenocorticotrophic hormone had a 75% success adrenocorticotrophic hormone had a 75% success raterate
After controlled trials were finally carried out, it was After controlled trials were finally carried out, it was found that 75% of infants return to normal without found that 75% of infants return to normal without treatmenttreatment
Identification of true cause of epidemic (oxygen to Identification of true cause of epidemic (oxygen to premature babies) was delayedpremature babies) was delayed
Dr Caroline Sabin, Royal Free HospitalDr Caroline Sabin, Royal Free Hospital UK-CAB - 13 August 2004- www.i-Base.infoUK-CAB - 13 August 2004- www.i-Base.info
Why do we need a control groupWhy do we need a control group
Uncontrolled trials may give a distorted view of a Uncontrolled trials may give a distorted view of a new therapynew therapy
Patients may improve over time, even without Patients may improve over time, even without treatment – thus, any improvement cannot treatment – thus, any improvement cannot necessarily be attributed to treatmentnecessarily be attributed to treatment
Patients selected for treatment may be less Patients selected for treatment may be less seriously ill than those not selected for treatment seriously ill than those not selected for treatment which may overestimate the benefits of new therapywhich may overestimate the benefits of new therapy
Patients in clinical trials generally do better than Patients in clinical trials generally do better than patients on same treatment who are not in trialspatients on same treatment who are not in trials
Dr Caroline Sabin, Royal Free HospitalDr Caroline Sabin, Royal Free Hospital UK-CAB - 13 August 2004- www.i-Base.infoUK-CAB - 13 August 2004- www.i-Base.info
Which control group? - the use of Which control group? - the use of historical or non-randomised controlshistorical or non-randomised controls
Controls less likely to have clearly defined Controls less likely to have clearly defined criteria for inclusion/exclusioncriteria for inclusion/exclusion
May have been a change in the type of patient May have been a change in the type of patient eligible for treatment, or prognosis may have eligible for treatment, or prognosis may have changed over timechanged over time
Investigator may have been more restrictive in Investigator may have been more restrictive in choice of patients for the trial, then when treating choice of patients for the trial, then when treating patients in the pastpatients in the past
Characteristics of patientsCharacteristics of patients
Dr Caroline Sabin, Royal Free HospitalDr Caroline Sabin, Royal Free Hospital UK-CAB - 13 August 2004- www.i-Base.infoUK-CAB - 13 August 2004- www.i-Base.info
Which control group? - the use of Which control group? - the use of historical or non-randomised controlshistorical or non-randomised controls
Quality of recorded data may not be as goodQuality of recorded data may not be as good
Definitions of response may differ between Definitions of response may differ between groups (eg. viral load endpoints)groups (eg. viral load endpoints)
Ancillary care may improve in a trial (eg. Ancillary care may improve in a trial (eg. adherence support, support for toxicities etc.)adherence support, support for toxicities etc.)
Experimental environmentExperimental environment
Thus, treatment and control groups may differ with Thus, treatment and control groups may differ with respect to many features other than treatment, and so respect to many features other than treatment, and so we cannot attribute any difference in outcome to the we cannot attribute any difference in outcome to the new treatmentnew treatment
Dr Caroline Sabin, Royal Free HospitalDr Caroline Sabin, Royal Free Hospital UK-CAB - 13 August 2004- www.i-Base.infoUK-CAB - 13 August 2004- www.i-Base.info
What is randomisation?What is randomisation?
Allocation of patients to treatments is determined Allocation of patients to treatments is determined by by chancechance
Randomised trials provide most Randomised trials provide most efficient efficient trial design trial design (ie. they are the most powerful) as they ensure that (ie. they are the most powerful) as they ensure that any factors that may affect outcome will be any factors that may affect outcome will be distributed equally between the treatment groupsdistributed equally between the treatment groups
This allows any difference in treatment response to This allows any difference in treatment response to be attributed to the treatmentbe attributed to the treatment
Removes impact of known confounding factors as Removes impact of known confounding factors as well as well as unknownunknown ones ones
Dr Caroline Sabin, Royal Free HospitalDr Caroline Sabin, Royal Free Hospital UK-CAB - 13 August 2004- www.i-Base.infoUK-CAB - 13 August 2004- www.i-Base.info
Why do trials need to be randomisedWhy do trials need to be randomised
Non-randomised trials have the potential to be Non-randomised trials have the potential to be seriously biasedseriously biased
If there are systematic differences between the If there are systematic differences between the patients in the treatment groups at the outset of patients in the treatment groups at the outset of the trial, then any differences in treatment the trial, then any differences in treatment response cannot necessarily be attributed to the response cannot necessarily be attributed to the new treatmentnew treatment
Eg. treatment comparisons in cohort studiesEg. treatment comparisons in cohort studies
Dr Caroline Sabin, Royal Free HospitalDr Caroline Sabin, Royal Free Hospital UK-CAB - 13 August 2004- www.i-Base.infoUK-CAB - 13 August 2004- www.i-Base.info
When can a randomised trial be doneWhen can a randomised trial be done
New treatment better than standard
New treatment worse than standard
‘Equipoise’
Who should have equipoise?Who should have equipoise?- The doctors recruiting patientsThe doctors recruiting patients- The patients entering the trialThe patients entering the trial
(is this true in reality?)(is this true in reality?)
Dr Caroline Sabin, Royal Free HospitalDr Caroline Sabin, Royal Free Hospital UK-CAB - 13 August 2004- www.i-Base.infoUK-CAB - 13 August 2004- www.i-Base.info
Other benefits of randomisationOther benefits of randomisation
Helps with blinding of trial (see later)Helps with blinding of trial (see later)
Prevents any conscious or subconscious Prevents any conscious or subconscious selection bias, whereby doctor tends to put more selection bias, whereby doctor tends to put more (or less) severely ill patients in a particular (or less) severely ill patients in a particular treatment grouptreatment group
Beware of any approach to randomisation Beware of any approach to randomisation whereby clinicians may be able to establish whereby clinicians may be able to establish treatment allocation prior to entry to the trial (eg. treatment allocation prior to entry to the trial (eg. systematic allocation by date of birth, alternate systematic allocation by date of birth, alternate allocation)allocation)
Dr Caroline Sabin, Royal Free HospitalDr Caroline Sabin, Royal Free Hospital UK-CAB - 13 August 2004- www.i-Base.infoUK-CAB - 13 August 2004- www.i-Base.info
Selection of patients for a trialSelection of patients for a trial
Discuss trial with patient and assess eligibility
Obtain informed consent
Formally enter patient into trial
Randomise
Dr Caroline Sabin, Royal Free HospitalDr Caroline Sabin, Royal Free Hospital UK-CAB - 13 August 2004- www.i-Base.infoUK-CAB - 13 August 2004- www.i-Base.info
Other benefits of randomisation (cont.)Other benefits of randomisation (cont.)
Example: Trial of anticoagulant therapy (Wright Example: Trial of anticoagulant therapy (Wright 1948)1948)
Patients admitted on odd days – anticoagulants Patients admitted on odd days – anticoagulants Patients admitted on even days – placeboPatients admitted on even days – placebo
Anticoagulant therapy – n=589Anticoagulant therapy – n=589Placebo – n=442Placebo – n=442
Dr Caroline Sabin, Royal Free HospitalDr Caroline Sabin, Royal Free Hospital UK-CAB - 13 August 2004- www.i-Base.infoUK-CAB - 13 August 2004- www.i-Base.info
The protocolThe protocol
Background, aims and objectives Trial design Patient selection – inclusion/exclusion criteria Treatment schedules Monitoring Registration, randomisation and blinding Methods of patient evaluation Patient consent Size of study Plans for dealing with protocol deviations Plans for statistical analysis Ethical approval and administrative matters
The ‘workshop manual’ for the trial. Will contain The ‘workshop manual’ for the trial. Will contain many or all of the following:many or all of the following:
Dr Caroline Sabin, Royal Free HospitalDr Caroline Sabin, Royal Free Hospital UK-CAB - 13 August 2004- www.i-Base.infoUK-CAB - 13 August 2004- www.i-Base.info
Selection of patients for a trialSelection of patients for a trial
A trial should have explicit inclusion criteria and A trial should have explicit inclusion criteria and exclusion criteria – precise definitions of who can exclusion criteria – precise definitions of who can be included in the studybe included in the study
Patients should be broadly representative of Patients should be broadly representative of some future group of patients to whom the trial some future group of patients to whom the trial may be appliedmay be applied
BUT – patients in trials are not necessarily a BUT – patients in trials are not necessarily a random selection of all HIV+ve individuals random selection of all HIV+ve individuals (unlikely to be the case)(unlikely to be the case)
Dr Caroline Sabin, Royal Free HospitalDr Caroline Sabin, Royal Free Hospital UK-CAB - 13 August 2004- www.i-Base.infoUK-CAB - 13 August 2004- www.i-Base.info
Evaluation of response – the primary Evaluation of response – the primary endpointendpoint
In any trial we need to define (preferably) a single In any trial we need to define (preferably) a single primary endpoint that captures the key effects of primary endpoint that captures the key effects of treatment on the patienttreatment on the patient
Primary endpoint is usually related to Primary endpoint is usually related to efficacyefficacy
If results from different endpoints are If results from different endpoints are inconsistent, the primary endpoint will be the one inconsistent, the primary endpoint will be the one on which any decisions about the value of the on which any decisions about the value of the drug will be mainly baseddrug will be mainly based
Dr Caroline Sabin, Royal Free HospitalDr Caroline Sabin, Royal Free Hospital UK-CAB - 13 August 2004- www.i-Base.infoUK-CAB - 13 August 2004- www.i-Base.info
Evaluation of response – secondary Evaluation of response – secondary endpointsendpoints
In addition to the primary endpoint, we may also In addition to the primary endpoint, we may also define any number of secondary endpointsdefine any number of secondary endpoints
These are often related to toxicity or quality of These are often related to toxicity or quality of life, or may be other measures of efficacy not life, or may be other measures of efficacy not captured by the primary endpointcaptured by the primary endpoint
Dr Caroline Sabin, Royal Free HospitalDr Caroline Sabin, Royal Free Hospital UK-CAB - 13 August 2004- www.i-Base.infoUK-CAB - 13 August 2004- www.i-Base.info
Definitions of endpoints – example Definitions of endpoints – example
Abacavir substitution for nucleoside analogs in Abacavir substitution for nucleoside analogs in patients with HIV lipoatrophy.patients with HIV lipoatrophy. Carr A et al. JAMA (2002); Carr A et al. JAMA (2002); 288: 207-215. 288: 207-215.
Primary endpoint:Primary endpoint:Mean change in limb fat mass measured by DXA at week 24Mean change in limb fat mass measured by DXA at week 24
Secondary endpoints:Secondary endpoints:Adverse eventsAdverse eventsAnthropometryAnthropometryTotal and central fat massTotal and central fat massBiochemical, lipid, and glycemic measurementsBiochemical, lipid, and glycemic measurementsViral loadViral loadCD4 countCD4 countQuality of lifeQuality of life
Dr Caroline Sabin, Royal Free HospitalDr Caroline Sabin, Royal Free Hospital UK-CAB - 13 August 2004- www.i-Base.infoUK-CAB - 13 August 2004- www.i-Base.info
Defining an endpointDefining an endpoint
In most trials patients are monitored very regularly In most trials patients are monitored very regularly (eg. every 4 weeks after randomisation(eg. every 4 weeks after randomisation
Tempting to compare treatments at each time point - Tempting to compare treatments at each time point - however, this is not advisable because of problems however, this is not advisable because of problems with multiple testing and the fact that the tests are with multiple testing and the fact that the tests are not independentnot independent
Thus, must select a single time point for assessment Thus, must select a single time point for assessment of the primary endpoint (eg. 24 weeks or 48 weeks)of the primary endpoint (eg. 24 weeks or 48 weeks)
Treatments should be formally compared at that Treatments should be formally compared at that timepoint only timepoint only
Dr Caroline Sabin, Royal Free HospitalDr Caroline Sabin, Royal Free Hospital UK-CAB - 13 August 2004- www.i-Base.infoUK-CAB - 13 August 2004- www.i-Base.info
Definitions of endpoints – example Definitions of endpoints – example
Abacavir substitution for nucleoside analogs in Abacavir substitution for nucleoside analogs in patients with HIV lipoatrophy.patients with HIV lipoatrophy. Carr A et al. JAMA (2002); Carr A et al. JAMA (2002); 288: 207-215. 288: 207-215.
Primary endpoint:Primary endpoint:Mean change in limb fat mass measured by DXA at week 24Mean change in limb fat mass measured by DXA at week 24
Secondary endpoints:Secondary endpoints:Adverse eventsAdverse eventsAnthropometryAnthropometryTotal and central fat massTotal and central fat massBiochemical, lipid, and glycemic measurementsBiochemical, lipid, and glycemic measurementsViral loadViral loadCD4 countCD4 countQuality of lifeQuality of life
Dr Caroline Sabin, Royal Free HospitalDr Caroline Sabin, Royal Free Hospital UK-CAB - 13 August 2004- www.i-Base.infoUK-CAB - 13 August 2004- www.i-Base.info
Clinical vs. surrogate endpointsClinical vs. surrogate endpoints
We are usually most interested in the effect of a new We are usually most interested in the effect of a new treatment on a clinical outcome (eg. new AIDS events treatment on a clinical outcome (eg. new AIDS events or death)or death)
However, currently, trials of HAART that use clinical However, currently, trials of HAART that use clinical endpoints generally have to be extremely large and endpoints generally have to be extremely large and follow patients for very long periods of time in order follow patients for very long periods of time in order to have sufficient power to detect a difference to have sufficient power to detect a difference between treatment regimensbetween treatment regimens
Thus, we often consider the effect of the treatment Thus, we often consider the effect of the treatment regimen on a surrogate endpoint (eg. change in CD4, regimen on a surrogate endpoint (eg. change in CD4, HIV RNA etc.)HIV RNA etc.)
Dr Caroline Sabin, Royal Free HospitalDr Caroline Sabin, Royal Free Hospital UK-CAB - 13 August 2004- www.i-Base.infoUK-CAB - 13 August 2004- www.i-Base.info
Surrogate endpointsSurrogate endpoints
““A laboratory measurement or a physical sign used A laboratory measurement or a physical sign used as a substitute for a clinically meaningful endpoint as a substitute for a clinically meaningful endpoint that measures directly how a patient feels, that measures directly how a patient feels, functions or survives.”functions or survives.”
Temple RJ. A regulatory authority’s opinion about surrogate endpoints. In: Temple RJ. A regulatory authority’s opinion about surrogate endpoints. In: Nimmo WS, Tucker GT, eds. Nimmo WS, Tucker GT, eds. Clinical measurement in drug evaluationClinical measurement in drug evaluation. . New York, NY: John Wiley & Sons Inc. 1995.New York, NY: John Wiley & Sons Inc. 1995.
Dr Caroline Sabin, Royal Free HospitalDr Caroline Sabin, Royal Free Hospital UK-CAB - 13 August 2004- www.i-Base.infoUK-CAB - 13 August 2004- www.i-Base.info
Surrogate endpoints (cont.)Surrogate endpoints (cont.)
In order for a laboratory marker to be a good In order for a laboratory marker to be a good surrogate endpoint for a clinical outcome, it has to surrogate endpoint for a clinical outcome, it has to fulfill two criteriafulfill two criteria
Surrogate must be on the Surrogate must be on the causal pathwaycausal pathway of the disease of the disease processprocess
Entire effect of the intervention on clinical outcome should Entire effect of the intervention on clinical outcome should be captured by changes in the surrogate be captured by changes in the surrogate
TreatmentTreatmentChanges in Changes in surrogatesurrogate
ImprovedImproved clinical outcomeclinical outcome
Dr Caroline Sabin, Royal Free HospitalDr Caroline Sabin, Royal Free Hospital UK-CAB - 13 August 2004- www.i-Base.infoUK-CAB - 13 August 2004- www.i-Base.info
Surrogate endpoints (cont.)Surrogate endpoints (cont.)
Pre-HAART, CD4 count was established as reliable Pre-HAART, CD4 count was established as reliable surrogate endpoint for AIDS/deathsurrogate endpoint for AIDS/death
Most trials now use HIV RNA as a surrogate Most trials now use HIV RNA as a surrogate endpoint (eg. viral load <50 copies/ml)endpoint (eg. viral load <50 copies/ml)
BUT – not all of the effect of the treatment (eg. BUT – not all of the effect of the treatment (eg. toxicities) may act through changes in the CD4 toxicities) may act through changes in the CD4 count or HIV RNA levelcount or HIV RNA level
Many combinations have similar virological Many combinations have similar virological efficacy – other outcomes may now be more efficacy – other outcomes may now be more importantimportant
Dr Caroline Sabin, Royal Free HospitalDr Caroline Sabin, Royal Free Hospital UK-CAB - 13 August 2004- www.i-Base.infoUK-CAB - 13 August 2004- www.i-Base.info
Definitions of endpoints – example Definitions of endpoints – example
Abacavir substitution for nucleoside analogs in Abacavir substitution for nucleoside analogs in patients with HIV lipoatrophy.patients with HIV lipoatrophy. Carr A et al. JAMA (2002); Carr A et al. JAMA (2002); 288: 207-215. 288: 207-215.
Primary endpoint:Primary endpoint:Mean change in limb fat mass measured by DXA at week 24Mean change in limb fat mass measured by DXA at week 24
Secondary endpoints:Secondary endpoints:Adverse eventsAdverse eventsAnthropometryAnthropometryTotal and central fat massTotal and central fat massBiochemical, lipid, and glycemic measurementsBiochemical, lipid, and glycemic measurementsViral loadViral loadCD4 countCD4 countQuality of lifeQuality of life
Dr Caroline Sabin, Royal Free HospitalDr Caroline Sabin, Royal Free Hospital UK-CAB - 13 August 2004- www.i-Base.infoUK-CAB - 13 August 2004- www.i-Base.info
‘‘Protocol violations’ Protocol violations’
Ineligible patientsIneligible patients – may be recruited by mistake – may be recruited by mistake
Non-adherent Non-adherent – may forget to take some or all of – may forget to take some or all of their drugs, may not attend for follow-up visits, their drugs, may not attend for follow-up visits, may take alternative treatmentsmay take alternative treatments
Patient withdrawalsPatient withdrawals – not able to tolerate drugs, – not able to tolerate drugs, may switch treatmentsmay switch treatments
For a number of reasons, patients included and For a number of reasons, patients included and randomised in the trial may not ‘behave’ as stated in randomised in the trial may not ‘behave’ as stated in the protocolthe protocol
QUESTION: how should these be dealt with in any QUESTION: how should these be dealt with in any analysis?analysis?
Dr Caroline Sabin, Royal Free HospitalDr Caroline Sabin, Royal Free Hospital UK-CAB - 13 August 2004- www.i-Base.infoUK-CAB - 13 August 2004- www.i-Base.info
Analysis by Intention-to-treat (ITT)Analysis by Intention-to-treat (ITT)
All patients randomised to treatment should be All patients randomised to treatment should be included in the analysis in the groups to which included in the analysis in the groups to which they were randomisedthey were randomised
Dr Caroline Sabin, Royal Free HospitalDr Caroline Sabin, Royal Free Hospital UK-CAB - 13 August 2004- www.i-Base.infoUK-CAB - 13 August 2004- www.i-Base.info
Analysis by Intention-to-treat (ITT)Analysis by Intention-to-treat (ITT)
Provides a measure of the real-life effect of Provides a measure of the real-life effect of treatmenttreatment
Is the Is the only unbiased estimateonly unbiased estimate of the of the treatment’s effecttreatment’s effect
Most major journals require analysis by ITT Most major journals require analysis by ITT
All presentations should include analysis by All presentations should include analysis by ITT as the primary analysis unless there is a ITT as the primary analysis unless there is a strong justification for not doing thisstrong justification for not doing this
Dr Caroline Sabin, Royal Free HospitalDr Caroline Sabin, Royal Free Hospital UK-CAB - 13 August 2004- www.i-Base.infoUK-CAB - 13 August 2004- www.i-Base.info
On-treatment analysesOn-treatment analyses
Only include those patients who complete a full Only include those patients who complete a full course of treatment to which they were course of treatment to which they were randomisedrandomised
Dr Caroline Sabin, Royal Free HospitalDr Caroline Sabin, Royal Free Hospital UK-CAB - 13 August 2004- www.i-Base.infoUK-CAB - 13 August 2004- www.i-Base.info
On-treatment analysesOn-treatment analyses
Suggested that this shows the optimal effect of Suggested that this shows the optimal effect of treatment when taken as recommendedtreatment when taken as recommended
However, has potential to provide However, has potential to provide extremely extremely biasedbiased estimates of treatment effect as those estimates of treatment effect as those with the worse responses to treatment are with the worse responses to treatment are likely to be the ones who drop-out/switch likely to be the ones who drop-out/switch treatmentstreatments
Approach will give an overly positive estimate Approach will give an overly positive estimate of effect of new treatmentof effect of new treatment
Dr Caroline Sabin, Royal Free HospitalDr Caroline Sabin, Royal Free Hospital UK-CAB - 13 August 2004- www.i-Base.infoUK-CAB - 13 August 2004- www.i-Base.info
On-treatment analyses - exampleOn-treatment analyses - example
RCT with primary endpoint of virological failure at RCT with primary endpoint of virological failure at week 48. Patients are allowed to switch therapy week 48. Patients are allowed to switch therapy once failure has occurred. once failure has occurred.
1
2
3
4
5
0 4 8 12 16 20 24 28 32 36 40 44 48
Weeks after randomisation
Patient number
CHANGED CHANGED TREATMENTTREATMENT
CHANGED CHANGED TREATMENTTREATMENT
CHANGED CHANGED TREATMENTTREATMENT
CHANGED CHANGED TREATMENTTREATMENT
Viral load > 50 copies/mlViral load > 50 copies/mlViral load Viral load << 50 copies/ml 50 copies/ml
Dr Caroline Sabin, Royal Free HospitalDr Caroline Sabin, Royal Free Hospital UK-CAB - 13 August 2004- www.i-Base.infoUK-CAB - 13 August 2004- www.i-Base.info
On-treatment analyses - exampleOn-treatment analyses - example
RCT with primary endpoint of virological failure at RCT with primary endpoint of virological failure at week 48. Patients are allowed to switch therapy week 48. Patients are allowed to switch therapy once failure has occurred. once failure has occurred.
1
2
3
4
5
0 4 8 12 16 20 24 28 32 36 40 44 48
Weeks after randomisation
Patient number
CHANGED CHANGED TREATMENTTREATMENT
CHANGED CHANGED TREATMENTTREATMENT
CHANGED CHANGED TREATMENTTREATMENT
CHANGED CHANGED TREATMENTTREATMENT
Viral load > 50 copies/mlViral load > 50 copies/mlViral load Viral load << 50 copies/ml 50 copies/ml
Primary endpoint Primary endpoint at week 48 = 1/1 at week 48 = 1/1 (100%)(100%)
Dr Caroline Sabin, Royal Free HospitalDr Caroline Sabin, Royal Free Hospital UK-CAB - 13 August 2004- www.i-Base.infoUK-CAB - 13 August 2004- www.i-Base.info
On-treatment analysesOn-treatment analyses
Those remaining on randomised treatment at Those remaining on randomised treatment at 48 weeks will, by definition, be those who have 48 weeks will, by definition, be those who have not experienced virological failurenot experienced virological failure
Anyone with virological failure prior to week 48 Anyone with virological failure prior to week 48 will change treatment and will be excluded will change treatment and will be excluded from the denominatorfrom the denominator
Primary event rate will always be close to 100% Primary event rate will always be close to 100% (depending on how quickly treatments are (depending on how quickly treatments are changed after virological failure)changed after virological failure)
FOR THIS REASON, ON-TREATMENT FOR THIS REASON, ON-TREATMENT ANALYSES SHOULD NOT BE USED FOR THE ANALYSES SHOULD NOT BE USED FOR THE PRIMARY ANALYSIS OF A TRIALPRIMARY ANALYSIS OF A TRIAL
Dr Caroline Sabin, Royal Free HospitalDr Caroline Sabin, Royal Free Hospital UK-CAB - 13 August 2004- www.i-Base.infoUK-CAB - 13 August 2004- www.i-Base.info
Problems when analysing by ITT with Problems when analysing by ITT with surrogate endpointssurrogate endpoints
If patients are lost-to-follow-up or drop out of a If patients are lost-to-follow-up or drop out of a trial, they are unlikely to attend for follow-up trial, they are unlikely to attend for follow-up visits and blood testsvisits and blood tests
Whilst it may be possible to obtain information Whilst it may be possible to obtain information on clinical endpoints from other sources, on clinical endpoints from other sources, information on CD4 counts or HIV RNA levels information on CD4 counts or HIV RNA levels may be unavailablemay be unavailable
Where data are missing, it is difficult to run a Where data are missing, it is difficult to run a ITT analysis in which all patients are included ITT analysis in which all patients are included in the analysisin the analysis
Dr Caroline Sabin, Royal Free HospitalDr Caroline Sabin, Royal Free Hospital UK-CAB - 13 August 2004- www.i-Base.infoUK-CAB - 13 August 2004- www.i-Base.info
Alternative methods of ITT analysesAlternative methods of ITT analyses
ITT Missing=Failure (ITT M=F)ITT Missing=Failure (ITT M=F)
All missing values are treated as failures in the All missing values are treated as failures in the analysis irrespective of most recent value – analysis irrespective of most recent value – ensures that all patients are included in the ensures that all patients are included in the denominator. If anything, this gives the most denominator. If anything, this gives the most pessimistic view of the new treatment.pessimistic view of the new treatment.
Where data on surrogate markers are missing, a Where data on surrogate markers are missing, a number of alternative strategies have been proposed:number of alternative strategies have been proposed:
Dr Caroline Sabin, Royal Free HospitalDr Caroline Sabin, Royal Free Hospital UK-CAB - 13 August 2004- www.i-Base.infoUK-CAB - 13 August 2004- www.i-Base.info
Alternative methods of ITT analysesAlternative methods of ITT analyses
ITT last observation carried forward (LOCF) ITT last observation carried forward (LOCF)
The last available measurement for each The last available measurement for each person is used in the analysis (irrespective of person is used in the analysis (irrespective of how long before the endpoint it was how long before the endpoint it was measured). This is an ITT analysis as all measured). This is an ITT analysis as all patients are included in the denominator but it patients are included in the denominator but it is not favoured by regulatory bodies (eg. FDA)is not favoured by regulatory bodies (eg. FDA)
Where data on surrogate markers are missing, a Where data on surrogate markers are missing, a number of alternative strategies have been proposed:number of alternative strategies have been proposed:
Dr Caroline Sabin, Royal Free HospitalDr Caroline Sabin, Royal Free Hospital UK-CAB - 13 August 2004- www.i-Base.infoUK-CAB - 13 August 2004- www.i-Base.info
Alternative methods of ITT analysesAlternative methods of ITT analyses
ITT missing=excludedITT missing=excluded
All patients with missing surrogate values are All patients with missing surrogate values are excluded from the analyses – this is NOT an excluded from the analyses – this is NOT an ITT analysis as the denominator does not ITT analysis as the denominator does not include all patients recruited to the trial. include all patients recruited to the trial. Essentially this is an Essentially this is an on-treatment analysison-treatment analysis
Where data on surrogate markers are missing, a Where data on surrogate markers are missing, a number of alternative strategies have been proposed:number of alternative strategies have been proposed:
Dr Caroline Sabin, Royal Free HospitalDr Caroline Sabin, Royal Free Hospital UK-CAB - 13 August 2004- www.i-Base.infoUK-CAB - 13 August 2004- www.i-Base.info
Examples of different approaches Examples of different approaches
123456789
1011121314151617181920
0 4 8 12 16 20 24 28 32 36 40 44 48
Weeks after randomisation
Patient number
Primary endpoint
Viral load > 50 copies/mlViral load > 50 copies/mlViral load Viral load << 50 copies/ml 50 copies/ml
Dr Caroline Sabin, Royal Free HospitalDr Caroline Sabin, Royal Free Hospital UK-CAB - 13 August 2004- www.i-Base.infoUK-CAB - 13 August 2004- www.i-Base.info
Examples of different approaches Examples of different approaches
123456789
1011121314151617181920
0 4 8 12 16 20 24 28 32 36 40 44 48
Weeks after randomisation
Patient number
Primary endpoint
Viral load > 50 copies/mlViral load > 50 copies/mlViral load Viral load << 50 copies/ml 50 copies/ml
Responder:Responder:On treatment/On treatment/ITT missingITT missing=excluded=excluded
111111----00----11--0011----11--11--1100
Response rate Response rate = 8/11= 8/11
= 73%= 73%
Dr Caroline Sabin, Royal Free HospitalDr Caroline Sabin, Royal Free Hospital UK-CAB - 13 August 2004- www.i-Base.infoUK-CAB - 13 August 2004- www.i-Base.info
Examples of different approaches Examples of different approaches
123456789
1011121314151617181920
0 4 8 12 16 20 24 28 32 36 40 44 48
Weeks after randomisation
Patient number
Primary endpoint
Viral load > 50 copies/mlViral load > 50 copies/mlViral load Viral load << 50 copies/ml 50 copies/ml
Responder:Responder:ITT missingITT missing
=failure=failure
1111110000000000110000110000110011001100
Response rate Response rate = 8/20= 8/20
= 40%= 40%
Dr Caroline Sabin, Royal Free HospitalDr Caroline Sabin, Royal Free Hospital UK-CAB - 13 August 2004- www.i-Base.infoUK-CAB - 13 August 2004- www.i-Base.info
Examples of different approaches Examples of different approaches
123456789
1011121314151617181920
0 4 8 12 16 20 24 28 32 36 40 44 48
Weeks after randomisation
Patient number
Primary endpoint
Viral load > 50 copies/mlViral load > 50 copies/mlViral load Viral load << 50 copies/ml 50 copies/ml
Responder:Responder:ITT missingITT missing
=LOCF=LOCF
1111110011000011111100110000110011001100
Response rate Response rate = 11/20= 11/20
= 55%= 55%
Dr Caroline Sabin, Royal Free HospitalDr Caroline Sabin, Royal Free Hospital UK-CAB - 13 August 2004- www.i-Base.infoUK-CAB - 13 August 2004- www.i-Base.info
Examples of different approaches - Examples of different approaches - summarysummary
ApproachApproach Response rateResponse rate
On treatment/ITT missing=excludedOn treatment/ITT missing=excluded 73%73%
ITT missing = failureITT missing = failure 40%40%
ITT missing = LOCFITT missing = LOCF 55%55%
Dr Caroline Sabin, Royal Free HospitalDr Caroline Sabin, Royal Free Hospital UK-CAB - 13 August 2004- www.i-Base.infoUK-CAB - 13 August 2004- www.i-Base.info
Subgroup analysesSubgroup analyses
It is often tempting to consider the effect of the It is often tempting to consider the effect of the treatment regimen in a number of subgroups of treatment regimen in a number of subgroups of the analysesthe analyses
For example, consider the effect of the regimen For example, consider the effect of the regimen in the following groups:in the following groups:
- Males/females - Males/females - Low/high viral load at baseline- Low/high viral load at baseline- Low/high CD4 count at baseline- Low/high CD4 count at baseline- ARV-naïve/ARV-experienced at start of trial- ARV-naïve/ARV-experienced at start of trial
Dr Caroline Sabin, Royal Free HospitalDr Caroline Sabin, Royal Free Hospital UK-CAB - 13 August 2004- www.i-Base.infoUK-CAB - 13 August 2004- www.i-Base.info
Subgroup analysesSubgroup analyses
There are a number of dangers inherent in There are a number of dangers inherent in performing too many subgroup analysesperforming too many subgroup analyses
The increased number of tests being The increased number of tests being performed means that there are problems of performed means that there are problems of multiple testing (ie. some of these multiple testing (ie. some of these comparisons are likely to be significant due to comparisons are likely to be significant due to chance)chance)
Although the study will have sufficient power Although the study will have sufficient power to detect a difference, the subgroups will often to detect a difference, the subgroups will often be based on a much smaller sample size and be based on a much smaller sample size and so will not be sufficiently poweredso will not be sufficiently powered
Dr Caroline Sabin, Royal Free HospitalDr Caroline Sabin, Royal Free Hospital UK-CAB - 13 August 2004- www.i-Base.infoUK-CAB - 13 August 2004- www.i-Base.info
Subgroup analyses – example 1Subgroup analyses – example 1
MalesMales FemalesFemales TotalTotal
AA BB AA BB AA BB
No. of patientsNo. of patients 110110 106106 2828 3030 138138 136136
No. (%) No. (%) RespondingResponding
77 (70%)77 (70%) 90 (85%)90 (85%) 20 (71%)20 (71%) 25 (83%)25 (83%) 97 (70%)97 (70%) 115 (85%)115 (85%)
P-value P-value
(A vs. B)(A vs. B)
0.010.01 0.440.44 0.0070.007
Although the difference between regimens A and B is Although the difference between regimens A and B is similar in women as it is in men, it is not significant due similar in women as it is in men, it is not significant due to the small number of women in the studyto the small number of women in the study
This This does not provide evidencedoes not provide evidence that there is no that there is no benefit of regimen B in womenbenefit of regimen B in women
Dr Caroline Sabin, Royal Free HospitalDr Caroline Sabin, Royal Free Hospital UK-CAB - 13 August 2004- www.i-Base.infoUK-CAB - 13 August 2004- www.i-Base.info
Subgroup analyses – example 2Subgroup analyses – example 2
MalesMales FemalesFemales TotalTotal
AA BB AA BB AA BB
No. of patientsNo. of patients 110110 106106 2828 3030 138138 136136
No. (%) No. (%) RespondingResponding
77 (70%)77 (70%) 86 (81%)86 (81%) 20 (71%)20 (71%) 29 (97%)29 (97%) 97 (70%)97 (70%) 115 (85%)115 (85%)
P-value P-value
(A vs. B)(A vs. B)
0.080.08 0.010.01 0.0070.007
Although regimen B now looks better in females than Although regimen B now looks better in females than males, a formal test of the interaction between sex and males, a formal test of the interaction between sex and treatment group (p=0.11), suggests that these results treatment group (p=0.11), suggests that these results are likely to have arisen by chanceare likely to have arisen by chance
Dr Caroline Sabin, Royal Free HospitalDr Caroline Sabin, Royal Free Hospital UK-CAB - 13 August 2004- www.i-Base.infoUK-CAB - 13 August 2004- www.i-Base.info
Subgroup analysesSubgroup analyses
In any trial analysis, if subgroup analyses are In any trial analysis, if subgroup analyses are thought to be important then they should be thought to be important then they should be specified specified a prioria priori in the protocol in the protocol
The study should be sufficiently large that The study should be sufficiently large that these subgroup analyses will be large enough these subgroup analyses will be large enough to detect important differencesto detect important differences
Evidence of a subgroup effect should never be Evidence of a subgroup effect should never be based on a comparison of p-values in the based on a comparison of p-values in the individual subgroups, but should be based on individual subgroups, but should be based on formal tests of formal tests of interactioninteraction between the factors between the factors of interestof interest
Dr Caroline Sabin, Royal Free HospitalDr Caroline Sabin, Royal Free Hospital UK-CAB - 13 August 2004- www.i-Base.infoUK-CAB - 13 August 2004- www.i-Base.info
Interim analysesInterim analyses
In any trial there is always a concern that one In any trial there is always a concern that one of the treatment arms may be inferior in some of the treatment arms may be inferior in some way to the others (eg. one regimen may be far way to the others (eg. one regimen may be far more efficacious or may be associated with a more efficacious or may be associated with a greater rate of serious toxicity than the others) greater rate of serious toxicity than the others)
If so, it may be considered to be ethically If so, it may be considered to be ethically unsound to continue to place patients at risk of unsound to continue to place patients at risk of the serious toxicity or of treatment failurethe serious toxicity or of treatment failure
As a result, one or more interim analyses may As a result, one or more interim analyses may be planned at pre-specified time points to be planned at pre-specified time points to monitor the progress of the trialmonitor the progress of the trial
Dr Caroline Sabin, Royal Free HospitalDr Caroline Sabin, Royal Free Hospital UK-CAB - 13 August 2004- www.i-Base.infoUK-CAB - 13 August 2004- www.i-Base.info
Interim analysesInterim analyses
However, there is always the chance that initial However, there is always the chance that initial findings, particularly on small numbers of findings, particularly on small numbers of patients, may have arisen by chancepatients, may have arisen by chance
If the trial is allowed to continue to completion, If the trial is allowed to continue to completion, these trends may disappearthese trends may disappear
Have to be very careful about stopping the trial Have to be very careful about stopping the trial early based on results of interim analysesearly based on results of interim analyses
If interim analyses are to be performed, then it If interim analyses are to be performed, then it is usually recommended that the trial is only is usually recommended that the trial is only stopped if evidence for a difference between stopped if evidence for a difference between the arms is very strong (eg. p<0.0001)the arms is very strong (eg. p<0.0001)
Dr Caroline Sabin, Royal Free HospitalDr Caroline Sabin, Royal Free Hospital UK-CAB - 13 August 2004- www.i-Base.infoUK-CAB - 13 August 2004- www.i-Base.info
Interim analyses – the role of a DSMBInterim analyses – the role of a DSMB
Often a Data Safety and Monitoring Board Often a Data Safety and Monitoring Board (DSMB) may be convened(DSMB) may be convened
Will include a number of independent ‘experts’ Will include a number of independent ‘experts’ in the area, usually including a statisticianin the area, usually including a statistician
The DSMB will evaluate safety data on a The DSMB will evaluate safety data on a regular basis (this information will not usually regular basis (this information will not usually be blinded) and will report back to the trial be blinded) and will report back to the trial Steering Committee Steering Committee
The DSMB may recommend that a trial be The DSMB may recommend that a trial be stopped early if necessarystopped early if necessary
Dr Caroline Sabin, Royal Free HospitalDr Caroline Sabin, Royal Free Hospital UK-CAB - 13 August 2004- www.i-Base.infoUK-CAB - 13 August 2004- www.i-Base.info
Interim analyses (cont)Interim analyses (cont)
If interim results suggest superiority of one of If interim results suggest superiority of one of the arms, but DSMB the arms, but DSMB do notdo not recommend recommend stopping the trial, presentation of results could stopping the trial, presentation of results could hinder the successful completion of the trialhinder the successful completion of the trial
Patients already randomised may switch to Patients already randomised may switch to superior arm, resulting in high levels of drop-outsuperior arm, resulting in high levels of drop-out
New patients will not wish to be randomised to the New patients will not wish to be randomised to the inferior arminferior arm
Dr Caroline Sabin, Royal Free HospitalDr Caroline Sabin, Royal Free Hospital UK-CAB - 13 August 2004- www.i-Base.infoUK-CAB - 13 August 2004- www.i-Base.info
Interim analyses (cont)Interim analyses (cont)
If data from interim analyses are to be If data from interim analyses are to be released, it is important that either blinding is released, it is important that either blinding is maintained, or results are not presented maintained, or results are not presented separately for the groupsseparately for the groups
In some cases, even blinded or combined data In some cases, even blinded or combined data may give an indication of the effect of the new may give an indication of the effect of the new drug/combination (e.g. in a placebo controlled drug/combination (e.g. in a placebo controlled trial)trial)
If this is the case, then no results concerning If this is the case, then no results concerning the primary endpoint of the trial (even blinded the primary endpoint of the trial (even blinded or combined) should be presentedor combined) should be presented
Dr Caroline Sabin, Royal Free HospitalDr Caroline Sabin, Royal Free Hospital UK-CAB - 13 August 2004- www.i-Base.infoUK-CAB - 13 August 2004- www.i-Base.info
Tests of superiorityTests of superiority
In a standard trial we usually test the null In a standard trial we usually test the null hypothesis that there is no difference between hypothesis that there is no difference between the treatment arms, against an alternative the treatment arms, against an alternative hypothesis that there is a difference between hypothesis that there is a difference between treatment arms treatment arms
Note that no direction is specified for this Note that no direction is specified for this difference (ie. drug A could be worse or better difference (ie. drug A could be worse or better than drug B)than drug B)
This is known as a This is known as a test of superioritytest of superiority, even , even though we don’t specify which drug is superiorthough we don’t specify which drug is superior
Dr Caroline Sabin, Royal Free HospitalDr Caroline Sabin, Royal Free Hospital UK-CAB - 13 August 2004- www.i-Base.infoUK-CAB - 13 August 2004- www.i-Base.info
Tests of equivalenceTests of equivalence
Sometimes, however, we may not want to test Sometimes, however, we may not want to test whether one drug is better than an another, but whether one drug is better than an another, but may simply want to show that the two drugs are may simply want to show that the two drugs are equivalentequivalent
This is usually the case when a new drug This is usually the case when a new drug appears to have similar efficacy but may have a appears to have similar efficacy but may have a better toxicity profile, be easier to take, or is better toxicity profile, be easier to take, or is cheapercheaper
Designing a study to show equivalence requires Designing a study to show equivalence requires a different emphasis to a study of superioritya different emphasis to a study of superiority
Dr Caroline Sabin, Royal Free HospitalDr Caroline Sabin, Royal Free Hospital UK-CAB - 13 August 2004- www.i-Base.infoUK-CAB - 13 August 2004- www.i-Base.info
-50
-40
-30
-20
-10
0
10
20
30
Tests of superiority – the effect of increasing Tests of superiority – the effect of increasing the sample sizethe sample size
Diff
eren
ce in
per
cent
age
Non-significant difference,but huge uncertainty
Similar difference but significant, due to increased
power of study
A more effective
A less effective
Dr Caroline Sabin, Royal Free HospitalDr Caroline Sabin, Royal Free Hospital UK-CAB - 13 August 2004- www.i-Base.infoUK-CAB - 13 August 2004- www.i-Base.info
Tests of equivalence (cont.)Tests of equivalence (cont.)
In a test of equivalence we focus more strongly In a test of equivalence we focus more strongly on the confidence interval for the treatment effecton the confidence interval for the treatment effect
The confidence interval around the treatment The confidence interval around the treatment effect must be narrow to exclude even a effect must be narrow to exclude even a moderate difference moderate difference
In order to do this, we usually require a much In order to do this, we usually require a much larger sample size than we would need to show larger sample size than we would need to show that one is superior to the otherthat one is superior to the other
Have to decide Have to decide a priori a priori on what can be deemed on what can be deemed as ‘equivalent’ as ‘equivalent’
Dr Caroline Sabin, Royal Free HospitalDr Caroline Sabin, Royal Free Hospital UK-CAB - 13 August 2004- www.i-Base.infoUK-CAB - 13 August 2004- www.i-Base.info
-30
-20
-10
0
10
20
Example – difference in percentage undetectable at Example – difference in percentage undetectable at 24 weeks with confidence interval (regimen A vs 24 weeks with confidence interval (regimen A vs regimen B) regimen B)
Diff
eren
ce in
per
cent
age
Non-significant difference,but huge uncertainty
Non-significant difference but less uncertainty
EQUIVALENCE RANGE
Dr Caroline Sabin, Royal Free HospitalDr Caroline Sabin, Royal Free Hospital UK-CAB - 13 August 2004- www.i-Base.infoUK-CAB - 13 August 2004- www.i-Base.info
Testing for equivalence (cont.)Testing for equivalence (cont.)
Need to specify the maximum amount by which it Need to specify the maximum amount by which it is thought that the two treatments could differ is thought that the two treatments could differ even when thought to be equivalenteven when thought to be equivalent
If the lower (or upper) limit of the CI of the If the lower (or upper) limit of the CI of the treatment effect does not exceed this value, then treatment effect does not exceed this value, then the two drugs are deemed equivalentthe two drugs are deemed equivalent
Sample size is chosen to ensure that the Sample size is chosen to ensure that the confidence interval around the treatment effect is confidence interval around the treatment effect is narrownarrow
Usually requires approximately twice as large a Usually requires approximately twice as large a sample as a test of non-equivalencesample as a test of non-equivalence
Dr Caroline Sabin, Royal Free HospitalDr Caroline Sabin, Royal Free Hospital UK-CAB - 13 August 2004- www.i-Base.infoUK-CAB - 13 August 2004- www.i-Base.info
Tests of non-inferiorityTests of non-inferiority
Conceptually similar to tests of equivalenceConceptually similar to tests of equivalence
New drug may be expected to be slightly inferior New drug may be expected to be slightly inferior to standard but at the same time offers other to standard but at the same time offers other benefits (eg. easier to take, less toxicities)benefits (eg. easier to take, less toxicities)
Need to show that the effect of the new treatment Need to show that the effect of the new treatment is not below some pre-stated is not below some pre-stated non-inferiority non-inferiority marginmargin
Confidence intervals again need to be narrow, Confidence intervals again need to be narrow, and sample sizes may be larger than in a and sample sizes may be larger than in a superiority trial superiority trial
Dr Caroline Sabin, Royal Free HospitalDr Caroline Sabin, Royal Free Hospital UK-CAB - 13 August 2004- www.i-Base.infoUK-CAB - 13 August 2004- www.i-Base.info
-30
-20
-10
0
10
20
Example – difference in percentage undetectable at Example – difference in percentage undetectable at 24 weeks with confidence interval (regimen A vs 24 weeks with confidence interval (regimen A vs regimen B) regimen B)
Diff
eren
ce in
per
cent
age
A more effective
A less effective
DRUG CONSIDERED INFERIOR
DRUG CONSIDERED NON-INFERIOR
Dr Caroline Sabin, Royal Free HospitalDr Caroline Sabin, Royal Free Hospital UK-CAB - 13 August 2004- www.i-Base.infoUK-CAB - 13 August 2004- www.i-Base.info
Group workGroup work
Dr Caroline Sabin, Royal Free HospitalDr Caroline Sabin, Royal Free Hospital UK-CAB - 13 August 2004- www.i-Base.infoUK-CAB - 13 August 2004- www.i-Base.info
Session 6: Critically appraising Session 6: Critically appraising researchresearch
Dr Caroline Sabin, Royal Free HospitalDr Caroline Sabin, Royal Free Hospital UK-CAB - 13 August 2004- www.i-Base.infoUK-CAB - 13 August 2004- www.i-Base.info
Why do we need to appraise researchWhy do we need to appraise research
Many people, particularly pharma companies, Many people, particularly pharma companies, have vested interests in some pieces of researchhave vested interests in some pieces of research
Although it is unlikely that anyone would Although it is unlikely that anyone would deliberately falsify research for their own deliberately falsify research for their own interests, the way in which results are presented interests, the way in which results are presented may be misleadingmay be misleading
Even if a study is perfectly carried out and Even if a study is perfectly carried out and presented appropriately, the results may not be presented appropriately, the results may not be applicable to your situationapplicable to your situation
Thus, we need to consider any piece of research Thus, we need to consider any piece of research carefully before acting on its recommendationscarefully before acting on its recommendations
Dr Caroline Sabin, Royal Free HospitalDr Caroline Sabin, Royal Free Hospital UK-CAB - 13 August 2004- www.i-Base.infoUK-CAB - 13 August 2004- www.i-Base.info
The peer-review process – journal articlesThe peer-review process – journal articles
Most major journals use a process known as Most major journals use a process known as peer-reviewpeer-review
Each submitted article is usually sent to two or Each submitted article is usually sent to two or more experts in the field so that they may give more experts in the field so that they may give their opinion on the design and conduct of the their opinion on the design and conduct of the study, the analytical methods used and study, the analytical methods used and importance of the resultsimportance of the results
On the basis of their reports, journals may either On the basis of their reports, journals may either reject the article, ask for a resubmission with reject the article, ask for a resubmission with changes, or accept the articlechanges, or accept the article
Dr Caroline Sabin, Royal Free HospitalDr Caroline Sabin, Royal Free Hospital UK-CAB - 13 August 2004- www.i-Base.infoUK-CAB - 13 August 2004- www.i-Base.info
Problems with the peer-review processProblems with the peer-review process
Hard to make the process truly blind – therefore Hard to make the process truly blind – therefore personal biases may be introducedpersonal biases may be introduced
May not be sent to someone who fully May not be sent to someone who fully understands or knows the areaunderstands or knows the area
Relies largely on goodwill of reviewers (payment, Relies largely on goodwill of reviewers (payment, if any, is minimal) and some put more time and if any, is minimal) and some put more time and effort into the task than otherseffort into the task than others
Thus, peer review system isn’t perfect and poor Thus, peer review system isn’t perfect and poor papers may be publishedpapers may be published
Difficult to improve the system though - having Difficult to improve the system though - having some system in place is better than none at all!some system in place is better than none at all!
Dr Caroline Sabin, Royal Free HospitalDr Caroline Sabin, Royal Free Hospital UK-CAB - 13 August 2004- www.i-Base.infoUK-CAB - 13 August 2004- www.i-Base.info
What is and isn’t publishedWhat is and isn’t published
Large studies are more likely to be published Large studies are more likely to be published than small ones, irrespective of study qualitythan small ones, irrespective of study quality
If the study is small then those studies that show If the study is small then those studies that show a significant result are more likely to be a significant result are more likely to be published (published (publication biaspublication bias))
There is a perception that if you are part of an There is a perception that if you are part of an established group with a known track record, it is established group with a known track record, it is easier to get things published – not sure whether easier to get things published – not sure whether this is backed up by evidence!this is backed up by evidence!
Dr Caroline Sabin, Royal Free HospitalDr Caroline Sabin, Royal Free Hospital UK-CAB - 13 August 2004- www.i-Base.infoUK-CAB - 13 August 2004- www.i-Base.info
Peer-review of other materialPeer-review of other material
Very limitedVery limited
Conference presentations are often selected on Conference presentations are often selected on basis of peer review of abstract onlybasis of peer review of abstract only
However, abstract is often very short, very vague However, abstract is often very short, very vague and may not contain all the information required and may not contain all the information required to make a valid decision on its qualityto make a valid decision on its quality
Conference abstracts are also selected to fit in Conference abstracts are also selected to fit in with the programme and planned sessionswith the programme and planned sessions
Dr Caroline Sabin, Royal Free HospitalDr Caroline Sabin, Royal Free Hospital UK-CAB - 13 August 2004- www.i-Base.infoUK-CAB - 13 August 2004- www.i-Base.info
The main questions when appraising The main questions when appraising research research
Do I believe the results?Do I believe the results?
Dr Caroline Sabin, Royal Free HospitalDr Caroline Sabin, Royal Free Hospital UK-CAB - 13 August 2004- www.i-Base.infoUK-CAB - 13 August 2004- www.i-Base.info
The main questions when appraising The main questions when appraising research research
Do I believe the results?Do I believe the results?
YESYES
Are the results important or new?Are the results important or new?
Dr Caroline Sabin, Royal Free HospitalDr Caroline Sabin, Royal Free Hospital UK-CAB - 13 August 2004- www.i-Base.infoUK-CAB - 13 August 2004- www.i-Base.info
The main questions when appraising The main questions when appraising research research
Do I believe the results?Do I believe the results?
YESYES
Are the results important or new?Are the results important or new?
Are the results applicable to me or to other Are the results applicable to me or to other people in a similar situation?people in a similar situation?
YESYES
Dr Caroline Sabin, Royal Free HospitalDr Caroline Sabin, Royal Free Hospital UK-CAB - 13 August 2004- www.i-Base.infoUK-CAB - 13 August 2004- www.i-Base.info
The main questions when appraising The main questions when appraising research research
Do I believe the results?Do I believe the results?
Dr Caroline Sabin, Royal Free HospitalDr Caroline Sabin, Royal Free Hospital UK-CAB - 13 August 2004- www.i-Base.infoUK-CAB - 13 August 2004- www.i-Base.info
Do I believe the results – are the results Do I believe the results – are the results valid? (RCTs)valid? (RCTs)
1.1. Was the assignment of patients to treatment Was the assignment of patients to treatment groups randomised? groups randomised?
a)a) How was the assignment list produced?How was the assignment list produced?
b)b) How was the assignment list concealed from the How was the assignment list concealed from the doctors?doctors?
Dr Caroline Sabin, Royal Free HospitalDr Caroline Sabin, Royal Free Hospital UK-CAB - 13 August 2004- www.i-Base.infoUK-CAB - 13 August 2004- www.i-Base.info
Do I believe the results – are the results Do I believe the results – are the results valid? (RCTs)valid? (RCTs)
2.2. Were all the patients who entered properly Were all the patients who entered properly accounted for?accounted for?
a)a) How “complete” was the follow-up?How “complete” was the follow-up?
b)b) How did the authors deal with patients who How did the authors deal with patients who did not receive assigned treatment or who did not receive assigned treatment or who deviated from the protocol?deviated from the protocol?
Was an ITT analysis performed?Was an ITT analysis performed?
Dr Caroline Sabin, Royal Free HospitalDr Caroline Sabin, Royal Free Hospital UK-CAB - 13 August 2004- www.i-Base.infoUK-CAB - 13 August 2004- www.i-Base.info
Do I believe the results – are the results Do I believe the results – are the results valid? (RCTs)valid? (RCTs)
3.3. To what extent was blinding carried out?To what extent was blinding carried out?
a)a) Patients?Patients?
b)b) Doctors?Doctors?
c)c) Other study personnel?Other study personnel?
Dr Caroline Sabin, Royal Free HospitalDr Caroline Sabin, Royal Free Hospital UK-CAB - 13 August 2004- www.i-Base.infoUK-CAB - 13 August 2004- www.i-Base.info
Do I believe the results – are the results Do I believe the results – are the results valid? (RCTs)valid? (RCTs)
4.4. How similar were groups at start of trial?How similar were groups at start of trial?
Dr Caroline Sabin, Royal Free HospitalDr Caroline Sabin, Royal Free Hospital UK-CAB - 13 August 2004- www.i-Base.infoUK-CAB - 13 August 2004- www.i-Base.info
Do I believe the results – are the results Do I believe the results – are the results valid? (RCTs)valid? (RCTs)
5.5. Aside from the experimental intervention, were Aside from the experimental intervention, were the two groups treated equally?the two groups treated equally?
Dr Caroline Sabin, Royal Free HospitalDr Caroline Sabin, Royal Free Hospital UK-CAB - 13 August 2004- www.i-Base.infoUK-CAB - 13 August 2004- www.i-Base.info
Assessing validity – some words of caution!Assessing validity – some words of caution!
Remember that it is very easy to criticise a paper, Remember that it is very easy to criticise a paper, but it is not always as easy to carry out the but it is not always as easy to carry out the research in the first placeresearch in the first place
It is very difficult to write the perfect paper It is very difficult to write the perfect paper (someone, somewhere will always find (someone, somewhere will always find something wrong with it)something wrong with it)
You have to decide if, putting all your criticisms You have to decide if, putting all your criticisms together, they are enough to make you seriously together, they are enough to make you seriously doubt the validity of the findings doubt the validity of the findings
Dr Caroline Sabin, Royal Free HospitalDr Caroline Sabin, Royal Free Hospital UK-CAB - 13 August 2004- www.i-Base.infoUK-CAB - 13 August 2004- www.i-Base.info
The main questions when appraising The main questions when appraising research research
Do I believe the results?Do I believe the results?
YESYES
Are the results important or new?Are the results important or new?
Dr Caroline Sabin, Royal Free HospitalDr Caroline Sabin, Royal Free Hospital UK-CAB - 13 August 2004- www.i-Base.infoUK-CAB - 13 August 2004- www.i-Base.info
Are the results important – what are the Are the results important – what are the results? (RCTs)results? (RCTs)
1.1. How large was the treatment effect?How large was the treatment effect?
Dr Caroline Sabin, Royal Free HospitalDr Caroline Sabin, Royal Free Hospital UK-CAB - 13 August 2004- www.i-Base.infoUK-CAB - 13 August 2004- www.i-Base.info
Are the results important – what are the Are the results important – what are the results? (RCTs)results? (RCTs)
1.1. How large was the treatment effect?How large was the treatment effect?
- Are the results clinically significant?- Are the results clinically significant?
- Relative risk? - Relative risk?
- Difference in risks? - Difference in risks?
- Number needed to treat? (an estimate of the - Number needed to treat? (an estimate of the number of patients who need to receive the number of patients who need to receive the treatment in order to prevent one ‘bad’ event)treatment in order to prevent one ‘bad’ event)
Dr Caroline Sabin, Royal Free HospitalDr Caroline Sabin, Royal Free Hospital UK-CAB - 13 August 2004- www.i-Base.infoUK-CAB - 13 August 2004- www.i-Base.info
Are the results important – what are the Are the results important – what are the results? (RCTs)results? (RCTs)
2.2. How precise was the treatment effect?How precise was the treatment effect?
Dr Caroline Sabin, Royal Free HospitalDr Caroline Sabin, Royal Free Hospital UK-CAB - 13 August 2004- www.i-Base.infoUK-CAB - 13 August 2004- www.i-Base.info
Are the results important – what are the Are the results important – what are the results? (RCTs)results? (RCTs)
2.2. How precise was the treatment effect?How precise was the treatment effect?
- How wide was the confidence interval? - How wide was the confidence interval?
- What interpretation do you make of the - What interpretation do you make of the confidence interval? confidence interval?
Dr Caroline Sabin, Royal Free HospitalDr Caroline Sabin, Royal Free Hospital UK-CAB - 13 August 2004- www.i-Base.infoUK-CAB - 13 August 2004- www.i-Base.info
The main questions when appraising The main questions when appraising research research
Do I believe the results?Do I believe the results?
YESYES
Are the results important or new?Are the results important or new?
Are the results applicable to me or to other Are the results applicable to me or to other people in a similar situation?people in a similar situation?
YESYES
Dr Caroline Sabin, Royal Free HospitalDr Caroline Sabin, Royal Free Hospital UK-CAB - 13 August 2004- www.i-Base.infoUK-CAB - 13 August 2004- www.i-Base.info
Are the results applicable to me? (RCTs)Are the results applicable to me? (RCTs)
1.1. Were patients in the trial similar to my own Were patients in the trial similar to my own situation?situation?
2.2. Did the authors consider all clinically important Did the authors consider all clinically important outcomes?outcomes?
3.3. Are the likely benefits of the new treatment worth Are the likely benefits of the new treatment worth the potential harms/costs?the potential harms/costs?
Dr Caroline Sabin, Royal Free HospitalDr Caroline Sabin, Royal Free Hospital UK-CAB - 13 August 2004- www.i-Base.infoUK-CAB - 13 August 2004- www.i-Base.info
Critically appraising cohort studiesCritically appraising cohort studies
General points are the same as for RCTs, although General points are the same as for RCTs, although clearly the issues of randomisation, blinding etc. are clearly the issues of randomisation, blinding etc. are not appropriatenot appropriate
Dr Caroline Sabin, Royal Free HospitalDr Caroline Sabin, Royal Free Hospital UK-CAB - 13 August 2004- www.i-Base.infoUK-CAB - 13 August 2004- www.i-Base.info
Critically appraising cohort studiesCritically appraising cohort studies
How representative is the cohort: How representative is the cohort:
Who is included in the cohortWho is included in the cohort
Who is excluded?Who is excluded?
Are there any differences between those included Are there any differences between those included and excluded that could limit the generalisability of and excluded that could limit the generalisability of the findings?the findings?
Dr Caroline Sabin, Royal Free HospitalDr Caroline Sabin, Royal Free Hospital UK-CAB - 13 August 2004- www.i-Base.infoUK-CAB - 13 August 2004- www.i-Base.info
Critically appraising cohort studiesCritically appraising cohort studies
Follow-up:Follow-up:
How is this maintained?How is this maintained?
How many patients are lost to follow-up?How many patients are lost to follow-up?
How are these patients dealt with in the analysis?How are these patients dealt with in the analysis?
Dr Caroline Sabin, Royal Free HospitalDr Caroline Sabin, Royal Free Hospital UK-CAB - 13 August 2004- www.i-Base.infoUK-CAB - 13 August 2004- www.i-Base.info
Critically appraising cohort studiesCritically appraising cohort studies
Temporal changes:Temporal changes:
If the authors are considering changes over time If the authors are considering changes over time (or some treatment whose use may change over (or some treatment whose use may change over time) then has anything else changed which could time) then has anything else changed which could explain the findings?explain the findings?
Dr Caroline Sabin, Royal Free HospitalDr Caroline Sabin, Royal Free Hospital UK-CAB - 13 August 2004- www.i-Base.infoUK-CAB - 13 August 2004- www.i-Base.info
Critically appraising cohort studiesCritically appraising cohort studies
Possible bias:Possible bias:
Have all other factors that could explain any Have all other factors that could explain any differences been considered?differences been considered?
Dr Caroline Sabin, Royal Free HospitalDr Caroline Sabin, Royal Free Hospital UK-CAB - 13 August 2004- www.i-Base.infoUK-CAB - 13 August 2004- www.i-Base.info
Group workGroup work