The Search for an Optimal Immunological Surrogate Endpoint...

119
Surrogate Endpoint Frameworks Optimal Surrogate Framework Simulation Studies Application to Dengue Trials Discussion The Search for an Optimal Immunological Surrogate Endpoint in Randomized Vaccine Efficacy Trials Peter Gilbert, Brenda Price, Mark van der Laan Sanofi Pasteur, Swiftwater, PA, September 26, 2018 Price B, Gilbert PB, van der Laan MJ. Estimation of the Optimal Surrogate Based on a Randomized Trial. 2018, Biometrics. 1 / 80

Transcript of The Search for an Optimal Immunological Surrogate Endpoint...

Page 1: The Search for an Optimal Immunological Surrogate Endpoint ...faculty.washington.edu/peterg/Sanofi2018/Slides/GilbertEstimated... · Surrogate Endpoint FrameworksOptimal Surrogate

Surrogate Endpoint Frameworks Optimal Surrogate Framework Simulation Studies Application to Dengue Trials Discussion

The Search for an Optimal ImmunologicalSurrogate Endpoint in Randomized Vaccine

Efficacy Trials

Peter Gilbert, Brenda Price, Mark van der Laan

Sanofi Pasteur, Swiftwater, PA, September 26, 2018

Price B, Gilbert PB, van der Laan MJ. Estimation of the OptimalSurrogate Based on a Randomized Trial. 2018, Biometrics.

1 / 80

Page 2: The Search for an Optimal Immunological Surrogate Endpoint ...faculty.washington.edu/peterg/Sanofi2018/Slides/GilbertEstimated... · Surrogate Endpoint FrameworksOptimal Surrogate

Surrogate Endpoint Frameworks Optimal Surrogate Framework Simulation Studies Application to Dengue Trials Discussion

Outline

1 Surrogate Endpoint Frameworks

2 Optimal Surrogate Framework

3 Simulation Studies

4 Application to Dengue VE Trials

5 Discussion

2 / 80

Page 3: The Search for an Optimal Immunological Surrogate Endpoint ...faculty.washington.edu/peterg/Sanofi2018/Slides/GilbertEstimated... · Surrogate Endpoint FrameworksOptimal Surrogate

Surrogate Endpoint Frameworks Optimal Surrogate Framework Simulation Studies Application to Dengue Trials Discussion

1 Surrogate Endpoint Frameworks

2 Optimal Surrogate Framework

3 Simulation Studies

4 Application to Dengue Trials

5 Discussion

3 / 80

Page 4: The Search for an Optimal Immunological Surrogate Endpoint ...faculty.washington.edu/peterg/Sanofi2018/Slides/GilbertEstimated... · Surrogate Endpoint FrameworksOptimal Surrogate

Surrogate Endpoint Frameworks Optimal Surrogate Framework Simulation Studies Application to Dengue Trials Discussion

Preventive Vaccine Efficacy Trial

07/18-20/2016 • 1

Preventive Vaccine Efficacy Trial

• Primary Objective

−Assess VE: Vaccine Efficacy to prevent infection or disease with a pathogen

• Secondary Objective

−Assess immune response biomarkers measured after vaccination as “immune correlates of protection” against infection or disease

Randomize

Vaccine

Measure immune response

Follow for clinical endpoint (Infection or Disease)

Receive inoculations

Placebo

Primary Objective

Assess vaccine efficacy (VE ) toprevent infection or disease witha pathogen

Secondary Objective

Assess immune responsebiomarkers measured aftervaccination as “surrogateendpoints”

4 / 80

Page 5: The Search for an Optimal Immunological Surrogate Endpoint ...faculty.washington.edu/peterg/Sanofi2018/Slides/GilbertEstimated... · Surrogate Endpoint FrameworksOptimal Surrogate

Surrogate Endpoint Frameworks Optimal Surrogate Framework Simulation Studies Application to Dengue Trials Discussion

Randomized Vaccine Efficacy Trial Notation

A −−−−−−−−−−−→ S −−−−−−−−−−−−−−−−−−−−−−−−−−−−−−→ Y

Fixed follow-up period of duration τ1

A = treatment (1=vaccine, 0=placebo or other control)

Y = clinical endpoint (1=event by τ1, 0=event-free at τ1)

S = candidate surrogate measured at time τ < τ1

Vaccine Efficacy Parameter

VE = 1− P(Y = 1|A = 1)

P(Y = 1|A = 0)

= Percent reduction in endpoint rate by vaccination

vs. control group

5 / 80

Page 6: The Search for an Optimal Immunological Surrogate Endpoint ...faculty.washington.edu/peterg/Sanofi2018/Slides/GilbertEstimated... · Surrogate Endpoint FrameworksOptimal Surrogate

Surrogate Endpoint Frameworks Optimal Surrogate Framework Simulation Studies Application to Dengue Trials Discussion

Randomized Vaccine Efficacy Trial Notation

A −−−−−−−−−−−→ S −−−−−−−−−−−−−−−−−−−−−−−−−−−−−−→ Y

Fixed follow-up period of duration τ1

A = treatment (1=vaccine, 0=placebo or other control)

Y = clinical endpoint (1=event by τ1, 0=event-free at τ1)

S = candidate surrogate measured at time τ < τ1

Vaccine Efficacy Parameter

VE = 1− P(Y = 1|A = 1)

P(Y = 1|A = 0)

= Percent reduction in endpoint rate by vaccination

vs. control group

5 / 80

Page 7: The Search for an Optimal Immunological Surrogate Endpoint ...faculty.washington.edu/peterg/Sanofi2018/Slides/GilbertEstimated... · Surrogate Endpoint FrameworksOptimal Surrogate

Surrogate Endpoint Frameworks Optimal Surrogate Framework Simulation Studies Application to Dengue Trials Discussion

Distinct “Surrogate” Frameworks (Price et al. Suppl.)

1 Prentice (1989) definition (Valid replacement endpoint)

Reliable inferences or predictions of VE based on S withoutusing Y

2 Controlled direct and indirect effects (Design newinterventions)

Predict VE for a new vaccine that sets S to certain levels

3 Natural direct and indirect effects (Mediation)

Insights into mechanisms/pathways of vaccine protection

4 VE curve (Response type effect modification)

Inferences on how VE varies over subgroups of vaccinerecipients defined by their response S to vaccination

5 Meta-analysis

Associate causal effects on S with causal effects on Y , forinference on VE in new settings

6 / 80

Page 8: The Search for an Optimal Immunological Surrogate Endpoint ...faculty.washington.edu/peterg/Sanofi2018/Slides/GilbertEstimated... · Surrogate Endpoint FrameworksOptimal Surrogate

Surrogate Endpoint Frameworks Optimal Surrogate Framework Simulation Studies Application to Dengue Trials Discussion

Distinct “Surrogate” Frameworks (Price et al. Suppl.)

1 Prentice (1989) definition (Valid replacement endpoint)

Reliable inferences or predictions of VE based on S withoutusing Y

2 Controlled direct and indirect effects (Design newinterventions)

Predict VE for a new vaccine that sets S to certain levels

3 Natural direct and indirect effects (Mediation)

Insights into mechanisms/pathways of vaccine protection

4 VE curve (Response type effect modification)

Inferences on how VE varies over subgroups of vaccinerecipients defined by their response S to vaccination

5 Meta-analysis

Associate causal effects on S with causal effects on Y , forinference on VE in new settings

6 / 80

Page 9: The Search for an Optimal Immunological Surrogate Endpoint ...faculty.washington.edu/peterg/Sanofi2018/Slides/GilbertEstimated... · Surrogate Endpoint FrameworksOptimal Surrogate

Surrogate Endpoint Frameworks Optimal Surrogate Framework Simulation Studies Application to Dengue Trials Discussion

Distinct “Surrogate” Frameworks (Price et al. Suppl.)

1 Prentice (1989) definition (Valid replacement endpoint)

Reliable inferences or predictions of VE based on S withoutusing Y

2 Controlled direct and indirect effects (Design newinterventions)

Predict VE for a new vaccine that sets S to certain levels

3 Natural direct and indirect effects (Mediation)

Insights into mechanisms/pathways of vaccine protection

4 VE curve (Response type effect modification)

Inferences on how VE varies over subgroups of vaccinerecipients defined by their response S to vaccination

5 Meta-analysis

Associate causal effects on S with causal effects on Y , forinference on VE in new settings

6 / 80

Page 10: The Search for an Optimal Immunological Surrogate Endpoint ...faculty.washington.edu/peterg/Sanofi2018/Slides/GilbertEstimated... · Surrogate Endpoint FrameworksOptimal Surrogate

Surrogate Endpoint Frameworks Optimal Surrogate Framework Simulation Studies Application to Dengue Trials Discussion

Distinct “Surrogate” Frameworks (Price et al. Suppl.)

1 Prentice (1989) definition (Valid replacement endpoint)

Reliable inferences or predictions of VE based on S withoutusing Y

2 Controlled direct and indirect effects (Design newinterventions)

Predict VE for a new vaccine that sets S to certain levels

3 Natural direct and indirect effects (Mediation)

Insights into mechanisms/pathways of vaccine protection

4 VE curve (Response type effect modification)

Inferences on how VE varies over subgroups of vaccinerecipients defined by their response S to vaccination

5 Meta-analysis

Associate causal effects on S with causal effects on Y , forinference on VE in new settings

6 / 80

Page 11: The Search for an Optimal Immunological Surrogate Endpoint ...faculty.washington.edu/peterg/Sanofi2018/Slides/GilbertEstimated... · Surrogate Endpoint FrameworksOptimal Surrogate

Surrogate Endpoint Frameworks Optimal Surrogate Framework Simulation Studies Application to Dengue Trials Discussion

Distinct “Surrogate” Frameworks (Price et al. Suppl.)

1 Prentice (1989) definition (Valid replacement endpoint)

Reliable inferences or predictions of VE based on S withoutusing Y

2 Controlled direct and indirect effects (Design newinterventions)

Predict VE for a new vaccine that sets S to certain levels

3 Natural direct and indirect effects (Mediation)

Insights into mechanisms/pathways of vaccine protection

4 VE curve (Response type effect modification)

Inferences on how VE varies over subgroups of vaccinerecipients defined by their response S to vaccination

5 Meta-analysis

Associate causal effects on S with causal effects on Y , forinference on VE in new settings 6 / 80

Page 12: The Search for an Optimal Immunological Surrogate Endpoint ...faculty.washington.edu/peterg/Sanofi2018/Slides/GilbertEstimated... · Surrogate Endpoint FrameworksOptimal Surrogate

Surrogate Endpoint Frameworks Optimal Surrogate Framework Simulation Studies Application to Dengue Trials Discussion

Typical Evaluation in VE Trials of Immune ResponseBiomarkers as Potential Surrogate Endpoints

In efficacy trials showing beneficial VE > 0, study selected immuneresponse biomarkers:

1 As correlates of risk (CoRs) of the disease endpoint in thevaccine and control groups

Extensive methods, e.g., adjusted association (Buyse et al.2000 Biostatistics)

2 For how well they adhere to criteria that imply the Prenticedefinition of a valid surrogate endpoint

Proportion of treatment effect explained (PTE) (Freedman etal. 1992 Stat Med)Proportion of treatment effect captured (Kobayashi andKuroki 2014 Stat Med)

3 As correlates of VE

Principal stratification modifier of VE (Follmann 2006Biometrics; Gilbert and Hudgens 2008 Biometrics)

7 / 80

Page 13: The Search for an Optimal Immunological Surrogate Endpoint ...faculty.washington.edu/peterg/Sanofi2018/Slides/GilbertEstimated... · Surrogate Endpoint FrameworksOptimal Surrogate

Surrogate Endpoint Frameworks Optimal Surrogate Framework Simulation Studies Application to Dengue Trials Discussion

Typical Evaluation in VE Trials of Immune ResponseBiomarkers as Potential Surrogate Endpoints

In efficacy trials showing beneficial VE > 0, study selected immuneresponse biomarkers:

1 As correlates of risk (CoRs) of the disease endpoint in thevaccine and control groups

Extensive methods, e.g., adjusted association (Buyse et al.2000 Biostatistics)

2 For how well they adhere to criteria that imply the Prenticedefinition of a valid surrogate endpoint

Proportion of treatment effect explained (PTE) (Freedman etal. 1992 Stat Med)Proportion of treatment effect captured (Kobayashi andKuroki 2014 Stat Med)

3 As correlates of VE

Principal stratification modifier of VE (Follmann 2006Biometrics; Gilbert and Hudgens 2008 Biometrics)

7 / 80

Page 14: The Search for an Optimal Immunological Surrogate Endpoint ...faculty.washington.edu/peterg/Sanofi2018/Slides/GilbertEstimated... · Surrogate Endpoint FrameworksOptimal Surrogate

Surrogate Endpoint Frameworks Optimal Surrogate Framework Simulation Studies Application to Dengue Trials Discussion

Typical Evaluation in VE Trials of Immune ResponseBiomarkers as Potential Surrogate Endpoints

In efficacy trials showing beneficial VE > 0, study selected immuneresponse biomarkers:

1 As correlates of risk (CoRs) of the disease endpoint in thevaccine and control groups

Extensive methods, e.g., adjusted association (Buyse et al.2000 Biostatistics)

2 For how well they adhere to criteria that imply the Prenticedefinition of a valid surrogate endpoint

Proportion of treatment effect explained (PTE) (Freedman etal. 1992 Stat Med)Proportion of treatment effect captured (Kobayashi andKuroki 2014 Stat Med)

3 As correlates of VE

Principal stratification modifier of VE (Follmann 2006Biometrics; Gilbert and Hudgens 2008 Biometrics)

7 / 80

Page 15: The Search for an Optimal Immunological Surrogate Endpoint ...faculty.washington.edu/peterg/Sanofi2018/Slides/GilbertEstimated... · Surrogate Endpoint FrameworksOptimal Surrogate

Surrogate Endpoint Frameworks Optimal Surrogate Framework Simulation Studies Application to Dengue Trials Discussion

The Approach Has Worked for Simple UnivariateSurrogates for Many Great Vaccines

Many licensed vaccines have excellent VE > 90%, and a singlemarker has been accepted by regulatory agencies as a surrogateendpoint (table from Norman Baylor, former FDA CBER director)

Regulatory Agencies Typically set Thresholds of Protection for Guiding Vaccine Licensure (this slide from Former FDA CBER Director, Dr. Norman Baylor)

Vaccine Test Correlate of Protection

Diphtheria Toxin Neutralization 0.01-0.1 IU/mL

Hepatitis A ELISA 10 mIU/mL

Hepatitis B ELISA 10 mIU/mL

Hib Polysaccharides ELISA 1 mcg/mL

Hib Conjugate ELISA 0.15 mcg/mL

Influenza HAI 1/40 dilution

Lyme ELISA 1100 EIA U/mL

Measles Microneutralization 120 mIU/mL

Pneumococcus ELISA (Opsonophagocytosis) 0.20-0.35 mcg/mL (for children);

1/8 dilution

Polio Serum Neutralization 1/4 - 1/8 dilution

Rabies Serum Neutralization 0.5 IU/mL

Rubella Immunoprecipitation 10-15 mIU/mL

Tetanus Toxin Neutralization 0.1 IU/mL

Varicella Serum Neutralization; gb ELISA 1/64 dilution 5 IU/mL

Adapted from Plotkin S. Correlates of Vaccine Induced Immunity (Vaccines 2008:47)

6/20/2013 • 1

8 / 80

Page 16: The Search for an Optimal Immunological Surrogate Endpoint ...faculty.washington.edu/peterg/Sanofi2018/Slides/GilbertEstimated... · Surrogate Endpoint FrameworksOptimal Surrogate

Surrogate Endpoint Frameworks Optimal Surrogate Framework Simulation Studies Application to Dengue Trials Discussion

However, In Our Era, More of a Multivariate LearningParadigm May be Helpful

Important pathogens for vaccine development (e.g., HIV,malaria, TB, influenza) have much greater genetic/antigenvariability than pathogens for which there is a great vaccine

Vastly more immune response biomarkers are now beingmeasured characterizing vaccine immunogenicity

Innate response systems vaccinology data (cellular subsets,transcriptomics, etc.) (high-dimensional)

Adaptive Response Data (Antibodies, T cells)

Binding antibody (Isotype, Subclass, Frequency, Magnitude,Breadth, Specificity)

Functional antibody (e.g., Neutralization, ADCC, Systemsserology) (Frequency, Magnitude, Breadth, Specificity)

CD4 and CD8 T cell responses (Frequency, Magnitude,Breadth, Specificity, Quality)

9 / 80

Page 17: The Search for an Optimal Immunological Surrogate Endpoint ...faculty.washington.edu/peterg/Sanofi2018/Slides/GilbertEstimated... · Surrogate Endpoint FrameworksOptimal Surrogate

Surrogate Endpoint Frameworks Optimal Surrogate Framework Simulation Studies Application to Dengue Trials Discussion

However, In Our Era, More of a Multivariate LearningParadigm May be Helpful

Important pathogens for vaccine development (e.g., HIV,malaria, TB, influenza) have much greater genetic/antigenvariability than pathogens for which there is a great vaccine

Vastly more immune response biomarkers are now beingmeasured characterizing vaccine immunogenicity

Innate response systems vaccinology data (cellular subsets,transcriptomics, etc.) (high-dimensional)

Adaptive Response Data (Antibodies, T cells)

Binding antibody (Isotype, Subclass, Frequency, Magnitude,Breadth, Specificity)

Functional antibody (e.g., Neutralization, ADCC, Systemsserology) (Frequency, Magnitude, Breadth, Specificity)

CD4 and CD8 T cell responses (Frequency, Magnitude,Breadth, Specificity, Quality)

9 / 80

Page 18: The Search for an Optimal Immunological Surrogate Endpoint ...faculty.washington.edu/peterg/Sanofi2018/Slides/GilbertEstimated... · Surrogate Endpoint FrameworksOptimal Surrogate

Surrogate Endpoint Frameworks Optimal Surrogate Framework Simulation Studies Application to Dengue Trials Discussion

However, In Our Era, More of a Multivariate LearningParadigm May be Helpful

Important pathogens for vaccine development (e.g., HIV,malaria, TB, influenza) have much greater genetic/antigenvariability than pathogens for which there is a great vaccine

Vastly more immune response biomarkers are now beingmeasured characterizing vaccine immunogenicity

Innate response systems vaccinology data (cellular subsets,transcriptomics, etc.) (high-dimensional)

Adaptive Response Data (Antibodies, T cells)

Binding antibody (Isotype, Subclass, Frequency, Magnitude,Breadth, Specificity)

Functional antibody (e.g., Neutralization, ADCC, Systemsserology) (Frequency, Magnitude, Breadth, Specificity)

CD4 and CD8 T cell responses (Frequency, Magnitude,Breadth, Specificity, Quality)

9 / 80

Page 19: The Search for an Optimal Immunological Surrogate Endpoint ...faculty.washington.edu/peterg/Sanofi2018/Slides/GilbertEstimated... · Surrogate Endpoint FrameworksOptimal Surrogate

Surrogate Endpoint Frameworks Optimal Surrogate Framework Simulation Studies Application to Dengue Trials Discussion

Example: Search for Immunological Surrogates in theRV144 HIV-1 Vaccine Efficacy TrialHIV-1 Vaccine Experience in the Assessment of Multiple Types of

Correlates for a Pox-Protein HIV-1 Vaccine [RV144 Thai Trial*]

T Cell Correlates Cytokine response (IL-10, IL-13) from Env stimulated PBMC

Polyfunctional CD4+ T cell (CD40L, IL-2, IL-4, IFN- and TNF-) and (CD40L, IL-2 and IL-4)

(Haynes et al. NEJM 2012; Lin et al. Nature Biotechnology 2015)

Host Genetics and AntibodiesIgG, IgG3, nAb, Avidity and FcRIICSNP IgA/ HLA A*02 allele IgA/ HLA II DQB1*06IgG/ HLA II DPB1*13

(Li et al. JCI 2014; Gartland et al. JV 2014; Prentice et al. Sci.Trans Med. 2015)

V2 Correlates V1V2 IgG, V1V2 IgG Breadth

V2 Linear AE hotspotV1V2 IgG3

(Haynes et al. NEJM 2012; Gottardo et al. Plos One 2013; Zolla-Pazner et al. Plos One 2014; Yates, Tomaras et al. Sci. Trans. Med 2014; Chung et al. Cell 2015)

IgA Correlates IgA Env Score IgA A. OOMSA gp140 CF IgA. A1 Congp140 IgA C1IgA Non-Vaccine StrainsIgA/IgG ratio

(Haynes et al. NEJM 2012; Tomaras, Ferrari et al. PNAS 2013)

Antibody Interaction CorrelatesLow IgA/ ADCC Low IgA/ nAbLow IgA/ IgG Env Avidity IgG3/ ADCCIgG3/IgG1

(Haynes et al. NEJM 2012; Tomaras, Ferrari et al. PNAS 2013; Yates et al. Sci. Trans. Med 2014; Chung et al. Cell 2015)

Virus Sieve Analysis and AntibodiesV2 Sieve (and V2 mAbs dependent on 169K)

Genetic distance from Vaccine strain /IgG and IgG3 V1V2 correlates

(Rolland, Edlefsen et al. Nature 2012; Liao et al. Immunity 2012; Gilbert et al. Statistics in Biosciences 2017)

Tomaras, Haynes (2014, Vaccines)Corey et al. (2015, Sci Transl Med)Tomaras, Plotkin (2017, Immunological Reviews)

*Rerks-Ngarm et al. (2009, NEJM)

0

10 / 80

Page 20: The Search for an Optimal Immunological Surrogate Endpoint ...faculty.washington.edu/peterg/Sanofi2018/Slides/GilbertEstimated... · Surrogate Endpoint FrameworksOptimal Surrogate

Surrogate Endpoint Frameworks Optimal Surrogate Framework Simulation Studies Application to Dengue Trials Discussion

Principles of the “Estimated Optimal Surrogate” Approachto Developing an Immunological Surrogate

Must be able to handle a high-dimensionality of immunologicalmeasurements, seeking to leverage all information in the databy modern computational machine learning

Yet still embrace the virtue of simplicity

In the end seek a simple univariate surrogate that is a synthesisand encapulation of all of the information

Make this simple surrogate clinically interpretable in terms ofvaccine efficacy

Set up the approach such that the excellent Prenticedefinition of a valid surrogate endpoint holds by construction

11 / 80

Page 21: The Search for an Optimal Immunological Surrogate Endpoint ...faculty.washington.edu/peterg/Sanofi2018/Slides/GilbertEstimated... · Surrogate Endpoint FrameworksOptimal Surrogate

Surrogate Endpoint Frameworks Optimal Surrogate Framework Simulation Studies Application to Dengue Trials Discussion

Principles of the “Estimated Optimal Surrogate” Approachto Developing an Immunological Surrogate

Must be able to handle a high-dimensionality of immunologicalmeasurements, seeking to leverage all information in the databy modern computational machine learning

Yet still embrace the virtue of simplicity

In the end seek a simple univariate surrogate that is a synthesisand encapulation of all of the information

Make this simple surrogate clinically interpretable in terms ofvaccine efficacy

Set up the approach such that the excellent Prenticedefinition of a valid surrogate endpoint holds by construction

11 / 80

Page 22: The Search for an Optimal Immunological Surrogate Endpoint ...faculty.washington.edu/peterg/Sanofi2018/Slides/GilbertEstimated... · Surrogate Endpoint FrameworksOptimal Surrogate

Surrogate Endpoint Frameworks Optimal Surrogate Framework Simulation Studies Application to Dengue Trials Discussion

Principles of the “Estimated Optimal Surrogate” Approachto Developing an Immunological Surrogate

Must be able to handle a high-dimensionality of immunologicalmeasurements, seeking to leverage all information in the databy modern computational machine learning

Yet still embrace the virtue of simplicity

In the end seek a simple univariate surrogate that is a synthesisand encapulation of all of the information

Make this simple surrogate clinically interpretable in terms ofvaccine efficacy

Set up the approach such that the excellent Prenticedefinition of a valid surrogate endpoint holds by construction

11 / 80

Page 23: The Search for an Optimal Immunological Surrogate Endpoint ...faculty.washington.edu/peterg/Sanofi2018/Slides/GilbertEstimated... · Surrogate Endpoint FrameworksOptimal Surrogate

Surrogate Endpoint Frameworks Optimal Surrogate Framework Simulation Studies Application to Dengue Trials Discussion

Principles of the “Estimated Optimal Surrogate” Approachto Developing an Immunological Surrogate

Must be able to handle a high-dimensionality of immunologicalmeasurements, seeking to leverage all information in the databy modern computational machine learning

Yet still embrace the virtue of simplicity

In the end seek a simple univariate surrogate that is a synthesisand encapulation of all of the information

Make this simple surrogate clinically interpretable in terms ofvaccine efficacy

Set up the approach such that the excellent Prenticedefinition of a valid surrogate endpoint holds by construction

11 / 80

Page 24: The Search for an Optimal Immunological Surrogate Endpoint ...faculty.washington.edu/peterg/Sanofi2018/Slides/GilbertEstimated... · Surrogate Endpoint FrameworksOptimal Surrogate

Surrogate Endpoint Frameworks Optimal Surrogate Framework Simulation Studies Application to Dengue Trials Discussion

Prentice (1989, Stat Med) Definition of a Valid Surrogate

VE = 1− P(Y = 1|A = 1)

P(Y = 1|A = 0)

Definition

S is a valid surrogate endpoint for Y if a valid test of

HY0 : No vaccine effect on Y (i.e., VE = 0)

is obtained by testing

HS0 : No vaccine effect on S

12 / 80

Page 25: The Search for an Optimal Immunological Surrogate Endpoint ...faculty.washington.edu/peterg/Sanofi2018/Slides/GilbertEstimated... · Surrogate Endpoint FrameworksOptimal Surrogate

Surrogate Endpoint Frameworks Optimal Surrogate Framework Simulation Studies Application to Dengue Trials Discussion

Illustration of Prentice Definition

LLOQ

10

102

103

104

105

Placebo Vaccine

Imm

une

resp

onse

bio

mar

ker

S

Corresponds to VE=0

LLOQ

10

102

103

104

105

Placebo Vaccine

Imm

une

resp

onse

bio

mar

ker

S

Corresponds to VE>0

13 / 80

Page 26: The Search for an Optimal Immunological Surrogate Endpoint ...faculty.washington.edu/peterg/Sanofi2018/Slides/GilbertEstimated... · Surrogate Endpoint FrameworksOptimal Surrogate

Surrogate Endpoint Frameworks Optimal Surrogate Framework Simulation Studies Application to Dengue Trials Discussion

The Surrogate Paradox

Virtue of the Prentice definition: Guarantees the surrogateparadox cannot occur

Surrogate Paradox∗

Postive vaccine effect on S

i.e., immune responses higher in vaccine than control group

S and Y are inversely correlated in both the vaccine andcontrol groups

i.e., in each group a higher immune response is associated witha lower disease rate

Yet VE < 0 (a harmful vaccine!)

∗E.g., Fleming and DeMets (1996, Ann Int Med), Chen et al. (2007,

JRSS-B), Ju and Geng (2010, JRSS-B), VanderWeele (2013,

Biometrics), Gilbert et al. (2015, J Causal Inference)

14 / 80

Page 27: The Search for an Optimal Immunological Surrogate Endpoint ...faculty.washington.edu/peterg/Sanofi2018/Slides/GilbertEstimated... · Surrogate Endpoint FrameworksOptimal Surrogate

Surrogate Endpoint Frameworks Optimal Surrogate Framework Simulation Studies Application to Dengue Trials Discussion

Example of the Surrogate Paradox

Sweden I Acellular Pertussis Trial of SKB and AventisPasteur vaccines vs. DT control arm (N ≈ 10, 000)∗

Immune response biomarkers S = FilamentousHaemagglutinin (FHA) and Pertussis Toxoid (PT) antibodyresponses higher for SKB than Aventis Pasteur vaccine

Higher FHA and PT antibodies associated with lowerpertussis disease rates

Yet estimated VE greater for the Aventis Pasteur vaccine:85% (95% CI 81–89%) vs. 58% (95% CI 51–66%)

Possible explanation: The Aventis Pasteur vaccine hadadditional antigens – Pertactin and Fimbriae types 2 and 3 –which stimulated additional immune responses contributing toprotection not measured by the FHA and PT assays

∗Gustafsson et al. (1996, NEJM); Fleming and Powers (2012, Stat Med)

15 / 80

Page 28: The Search for an Optimal Immunological Surrogate Endpoint ...faculty.washington.edu/peterg/Sanofi2018/Slides/GilbertEstimated... · Surrogate Endpoint FrameworksOptimal Surrogate

Surrogate Endpoint Frameworks Optimal Surrogate Framework Simulation Studies Application to Dengue Trials Discussion

Example of the Surrogate Paradox

Sweden I Acellular Pertussis Trial of SKB and AventisPasteur vaccines vs. DT control arm (N ≈ 10, 000)∗

Immune response biomarkers S = FilamentousHaemagglutinin (FHA) and Pertussis Toxoid (PT) antibodyresponses higher for SKB than Aventis Pasteur vaccine

Higher FHA and PT antibodies associated with lowerpertussis disease rates

Yet estimated VE greater for the Aventis Pasteur vaccine:85% (95% CI 81–89%) vs. 58% (95% CI 51–66%)

Possible explanation: The Aventis Pasteur vaccine hadadditional antigens – Pertactin and Fimbriae types 2 and 3 –which stimulated additional immune responses contributing toprotection not measured by the FHA and PT assays

∗Gustafsson et al. (1996, NEJM); Fleming and Powers (2012, Stat Med)

15 / 80

Page 29: The Search for an Optimal Immunological Surrogate Endpoint ...faculty.washington.edu/peterg/Sanofi2018/Slides/GilbertEstimated... · Surrogate Endpoint FrameworksOptimal Surrogate

Surrogate Endpoint Frameworks Optimal Surrogate Framework Simulation Studies Application to Dengue Trials Discussion

Example of the Surrogate Paradox

Sweden I Acellular Pertussis Trial of SKB and AventisPasteur vaccines vs. DT control arm (N ≈ 10, 000)∗

Immune response biomarkers S = FilamentousHaemagglutinin (FHA) and Pertussis Toxoid (PT) antibodyresponses higher for SKB than Aventis Pasteur vaccine

Higher FHA and PT antibodies associated with lowerpertussis disease rates

Yet estimated VE greater for the Aventis Pasteur vaccine:85% (95% CI 81–89%) vs. 58% (95% CI 51–66%)

Possible explanation: The Aventis Pasteur vaccine hadadditional antigens – Pertactin and Fimbriae types 2 and 3 –which stimulated additional immune responses contributing toprotection not measured by the FHA and PT assays

∗Gustafsson et al. (1996, NEJM); Fleming and Powers (2012, Stat Med)15 / 80

Page 30: The Search for an Optimal Immunological Surrogate Endpoint ...faculty.washington.edu/peterg/Sanofi2018/Slides/GilbertEstimated... · Surrogate Endpoint FrameworksOptimal Surrogate

Surrogate Endpoint Frameworks Optimal Surrogate Framework Simulation Studies Application to Dengue Trials Discussion

1 Surrogate Endpoint Frameworks

2 Optimal Surrogate Framework

3 Simulation Studies

4 Application to Dengue Trials

5 Discussion

16 / 80

Page 31: The Search for an Optimal Immunological Surrogate Endpoint ...faculty.washington.edu/peterg/Sanofi2018/Slides/GilbertEstimated... · Surrogate Endpoint FrameworksOptimal Surrogate

Surrogate Endpoint Frameworks Optimal Surrogate Framework Simulation Studies Application to Dengue Trials Discussion

Introduction to an Optimal Surrogate

Data from a VE Trial for Developing a Surrogate

W = Baseline covariates

A = Randomized treatment assignment(1=vaccine, 0=placebo)

S = Immune response biomarkers measured by anintermediate time point τ (e.g., 2 weeks post last vaccination)

Y = Disease endpoint by the end of follow-up τ1 after τ

Goal: Develop a most-promising surrogate endpoint for the diseaseendpoint so that future randomized studies can restrict themselvesto only collecting the surrogate outcome

17 / 80

Page 32: The Search for an Optimal Immunological Surrogate Endpoint ...faculty.washington.edu/peterg/Sanofi2018/Slides/GilbertEstimated... · Surrogate Endpoint FrameworksOptimal Surrogate

Surrogate Endpoint Frameworks Optimal Surrogate Framework Simulation Studies Application to Dengue Trials Discussion

Optimal Surrogate = Valid Surrogate that OptimallyPredicts Y

Define an optimal surrogate as the function of (W ,A, S)that satisfies the Prentice definition and that optimallypredicts Y

A true (unknown) parameter that is estimated

Goal 1: Obtain an efficient and robust estimate of theoptimal surrogate based on the randomized efficacy trial

Goal 2: Use the estimated optimal surrogate built for Goal 1in future clinical trials for estimation and testing of VE(treatment effect on Y )

Tackles the bridging objective of inferring the causaltreatment effect VE in a new trial without measuring Y(also addressed by Pearl and Bareinboim, 2011, 2012)

18 / 80

Page 33: The Search for an Optimal Immunological Surrogate Endpoint ...faculty.washington.edu/peterg/Sanofi2018/Slides/GilbertEstimated... · Surrogate Endpoint FrameworksOptimal Surrogate

Surrogate Endpoint Frameworks Optimal Surrogate Framework Simulation Studies Application to Dengue Trials Discussion

Optimal Surrogate = Valid Surrogate that OptimallyPredicts Y

Define an optimal surrogate as the function of (W ,A, S)that satisfies the Prentice definition and that optimallypredicts Y

A true (unknown) parameter that is estimated

Goal 1: Obtain an efficient and robust estimate of theoptimal surrogate based on the randomized efficacy trial

Goal 2: Use the estimated optimal surrogate built for Goal 1in future clinical trials for estimation and testing of VE(treatment effect on Y )

Tackles the bridging objective of inferring the causaltreatment effect VE in a new trial without measuring Y(also addressed by Pearl and Bareinboim, 2011, 2012)

18 / 80

Page 34: The Search for an Optimal Immunological Surrogate Endpoint ...faculty.washington.edu/peterg/Sanofi2018/Slides/GilbertEstimated... · Surrogate Endpoint FrameworksOptimal Surrogate

Surrogate Endpoint Frameworks Optimal Surrogate Framework Simulation Studies Application to Dengue Trials Discussion

Optimal Surrogate = Valid Surrogate that OptimallyPredicts Y

Define an optimal surrogate as the function of (W ,A, S)that satisfies the Prentice definition and that optimallypredicts Y

A true (unknown) parameter that is estimated

Goal 1: Obtain an efficient and robust estimate of theoptimal surrogate based on the randomized efficacy trial

Goal 2: Use the estimated optimal surrogate built for Goal 1in future clinical trials for estimation and testing of VE(treatment effect on Y )

Tackles the bridging objective of inferring the causaltreatment effect VE in a new trial without measuring Y(also addressed by Pearl and Bareinboim, 2011, 2012)

18 / 80

Page 35: The Search for an Optimal Immunological Surrogate Endpoint ...faculty.washington.edu/peterg/Sanofi2018/Slides/GilbertEstimated... · Surrogate Endpoint FrameworksOptimal Surrogate

Surrogate Endpoint Frameworks Optimal Surrogate Framework Simulation Studies Application to Dengue Trials Discussion

Introduction to an Optimal Surrogate

This work is about the search for promising surrogatesbased on an efficacy trial(s) with (W ,A,S ,Y ) measured

A promising surrogate is one that satisfies the Prenticedefinition and is optimally predictive of Y in this originaltrial(s)

This is a good starting point for building a surrogate that ispromising for the ultimate objective of bridging – inference onVE in new settings based on (W ,A,S)

19 / 80

Page 36: The Search for an Optimal Immunological Surrogate Endpoint ...faculty.washington.edu/peterg/Sanofi2018/Slides/GilbertEstimated... · Surrogate Endpoint FrameworksOptimal Surrogate

Surrogate Endpoint Frameworks Optimal Surrogate Framework Simulation Studies Application to Dengue Trials Discussion

Applications of the Estimated Optimal Surrogate Approachto Immunological Surrogate Development

1 A given immune response biomarker is thought to provide asufficiently valid surrogate endpoint, but it is unclear how tooptimally define the readout

E.g., the CYD14 and CYD15 dengue phase 3 VE trials studiedPRNT50, a single estimated summary measure from astatistical model fit to a neutralization dilution series curveIs there a better surrogate based on a different feature of thecurve? Would an alternative neutralization assay do better(e.g., Microneutralization Version 2)?

2 Additional assays are applied measuring new immune responsefeatures (e.g., Fc effector function assays, T cell assays, innateimmunity assays) and we ask whether an improved surrogatecan be developed by adding one or more assays?

E.g., in RV144, the original anti-V2 antibody correlate of riskwas improved by adding ADCC and CD4 T cellpolyfunctionality (Haynes et al., 2012, NEJM; Lin et al., 2015,Nat Biotech)

20 / 80

Page 37: The Search for an Optimal Immunological Surrogate Endpoint ...faculty.washington.edu/peterg/Sanofi2018/Slides/GilbertEstimated... · Surrogate Endpoint FrameworksOptimal Surrogate

Surrogate Endpoint Frameworks Optimal Surrogate Framework Simulation Studies Application to Dengue Trials Discussion

Applications of the Estimated Optimal Surrogate Approachto Immunological Surrogate Development

1 A given immune response biomarker is thought to provide asufficiently valid surrogate endpoint, but it is unclear how tooptimally define the readout

E.g., the CYD14 and CYD15 dengue phase 3 VE trials studiedPRNT50, a single estimated summary measure from astatistical model fit to a neutralization dilution series curveIs there a better surrogate based on a different feature of thecurve? Would an alternative neutralization assay do better(e.g., Microneutralization Version 2)?

2 Additional assays are applied measuring new immune responsefeatures (e.g., Fc effector function assays, T cell assays, innateimmunity assays) and we ask whether an improved surrogatecan be developed by adding one or more assays?

E.g., in RV144, the original anti-V2 antibody correlate of riskwas improved by adding ADCC and CD4 T cellpolyfunctionality (Haynes et al., 2012, NEJM; Lin et al., 2015,Nat Biotech) 20 / 80

Page 38: The Search for an Optimal Immunological Surrogate Endpoint ...faculty.washington.edu/peterg/Sanofi2018/Slides/GilbertEstimated... · Surrogate Endpoint FrameworksOptimal Surrogate

Surrogate Endpoint Frameworks Optimal Surrogate Framework Simulation Studies Application to Dengue Trials Discussion

Applications of the Estimated Optimal Surrogate Approachto Immunological Surrogate Development, Continued

3 At the outset of a correlates study a set of (possiblyhigh-dimensional) immune response biomarkers are measured,and we wish to develop best surrogates based on this set

Currently planning such an analysis for the first TB vaccineinfection endpoint efficacy trial (Nemes et al., 2018, NEJM)

4 Immune response assays are measured at multiple time points(e.g., baseline and post vaccinations, possibly longitudinally),and we wish to study whether a surrogate can be improved byincluding mutiple time points

E.g., in the dengue trials, accounting for both baseline(pre-existing immunity) and post-vaccination readouts isevidently important (Moodie et al., 2018, J Infect Dis; Sridharet al., 2018, NEJM)

For each application, a principled framework is needed forestimating optimal surrogates and for comparing theperformance of different estimators

21 / 80

Page 39: The Search for an Optimal Immunological Surrogate Endpoint ...faculty.washington.edu/peterg/Sanofi2018/Slides/GilbertEstimated... · Surrogate Endpoint FrameworksOptimal Surrogate

Surrogate Endpoint Frameworks Optimal Surrogate Framework Simulation Studies Application to Dengue Trials Discussion

Applications of the Estimated Optimal Surrogate Approachto Immunological Surrogate Development, Continued

3 At the outset of a correlates study a set of (possiblyhigh-dimensional) immune response biomarkers are measured,and we wish to develop best surrogates based on this set

Currently planning such an analysis for the first TB vaccineinfection endpoint efficacy trial (Nemes et al., 2018, NEJM)

4 Immune response assays are measured at multiple time points(e.g., baseline and post vaccinations, possibly longitudinally),and we wish to study whether a surrogate can be improved byincluding mutiple time points

E.g., in the dengue trials, accounting for both baseline(pre-existing immunity) and post-vaccination readouts isevidently important (Moodie et al., 2018, J Infect Dis; Sridharet al., 2018, NEJM)

For each application, a principled framework is needed forestimating optimal surrogates and for comparing theperformance of different estimators

21 / 80

Page 40: The Search for an Optimal Immunological Surrogate Endpoint ...faculty.washington.edu/peterg/Sanofi2018/Slides/GilbertEstimated... · Surrogate Endpoint FrameworksOptimal Surrogate

Surrogate Endpoint Frameworks Optimal Surrogate Framework Simulation Studies Application to Dengue Trials Discussion

Applications of the Estimated Optimal Surrogate Approachto Immunological Surrogate Development, Continued

3 At the outset of a correlates study a set of (possiblyhigh-dimensional) immune response biomarkers are measured,and we wish to develop best surrogates based on this set

Currently planning such an analysis for the first TB vaccineinfection endpoint efficacy trial (Nemes et al., 2018, NEJM)

4 Immune response assays are measured at multiple time points(e.g., baseline and post vaccinations, possibly longitudinally),and we wish to study whether a surrogate can be improved byincluding mutiple time points

E.g., in the dengue trials, accounting for both baseline(pre-existing immunity) and post-vaccination readouts isevidently important (Moodie et al., 2018, J Infect Dis; Sridharet al., 2018, NEJM)

For each application, a principled framework is needed forestimating optimal surrogates and for comparing theperformance of different estimators

21 / 80

Page 41: The Search for an Optimal Immunological Surrogate Endpoint ...faculty.washington.edu/peterg/Sanofi2018/Slides/GilbertEstimated... · Surrogate Endpoint FrameworksOptimal Surrogate

Surrogate Endpoint Frameworks Optimal Surrogate Framework Simulation Studies Application to Dengue Trials Discussion

Statistical Formulation of an Optimal Surrogate

Data from a VE Trial for Developing a Surrogate

W = Baseline covariates

A = Randomized treatment assignment(1=vaccine, 0=placebo)

S = Immune response biomarkers measured by anintermediate time point τ (e.g., 2 weeks post last vaccination)

Y = Disease endpoint by the end of follow-up τ1 after τ

Case-cohort or case-control sampling design, where S (and perhapscomponents of W ) is measured in a subset of study participants

22 / 80

Page 42: The Search for an Optimal Immunological Surrogate Endpoint ...faculty.washington.edu/peterg/Sanofi2018/Slides/GilbertEstimated... · Surrogate Endpoint FrameworksOptimal Surrogate

Surrogate Endpoint Frameworks Optimal Surrogate Framework Simulation Studies Application to Dengue Trials Discussion

A Nonparametric, Robust Approach

Historically, evaluating surrogates has relied on correctlyspecified regression models linking disease risk to inputcovariates (A,W ,S)

E.g., logistic regression or Cox regressionMis-specified models leads to biased estimation and potentiallymisleading results about surrogate endpoints

This nonparametric approach avoids assumptions on thedistribution of W or on the conditional distribution of (S ,Y )given A,W , and thus is more robust

23 / 80

Page 43: The Search for an Optimal Immunological Surrogate Endpoint ...faculty.washington.edu/peterg/Sanofi2018/Slides/GilbertEstimated... · Surrogate Endpoint FrameworksOptimal Surrogate

Surrogate Endpoint Frameworks Optimal Surrogate Framework Simulation Studies Application to Dengue Trials Discussion

A Nonparametric, Robust Approach

Historically, evaluating surrogates has relied on correctlyspecified regression models linking disease risk to inputcovariates (A,W ,S)

E.g., logistic regression or Cox regressionMis-specified models leads to biased estimation and potentiallymisleading results about surrogate endpoints

This nonparametric approach avoids assumptions on thedistribution of W or on the conditional distribution of (S ,Y )given A,W , and thus is more robust

23 / 80

Page 44: The Search for an Optimal Immunological Surrogate Endpoint ...faculty.washington.edu/peterg/Sanofi2018/Slides/GilbertEstimated... · Surrogate Endpoint FrameworksOptimal Surrogate

Surrogate Endpoint Frameworks Optimal Surrogate Framework Simulation Studies Application to Dengue Trials Discussion

Candidate Surrogate Outcomes (True UnknownParameters)

Any real-valued function (W ,A, S)→ ψ(W ,A,S) is acandidate surrogate, representing a measurement one cancollect by time τ and depending on the unknown trueobserved data distribution P0

Question: How to define a good surrogate in terms of P0?

Starting point: Only consider Sψ ≡ ψ(W ,A, S) that arevalid in the efficacy study, according to the Prentice definition:

VE = 0 ⇐⇒ Vaccine effect on the mean of Sψ = 0

24 / 80

Page 45: The Search for an Optimal Immunological Surrogate Endpoint ...faculty.washington.edu/peterg/Sanofi2018/Slides/GilbertEstimated... · Surrogate Endpoint FrameworksOptimal Surrogate

Surrogate Endpoint Frameworks Optimal Surrogate Framework Simulation Studies Application to Dengue Trials Discussion

Candidate Surrogate Outcomes (True UnknownParameters)

Any real-valued function (W ,A, S)→ ψ(W ,A,S) is acandidate surrogate, representing a measurement one cancollect by time τ and depending on the unknown trueobserved data distribution P0

Question: How to define a good surrogate in terms of P0?

Starting point: Only consider Sψ ≡ ψ(W ,A, S) that arevalid in the efficacy study, according to the Prentice definition:

VE = 0 ⇐⇒ Vaccine effect on the mean of Sψ = 0

24 / 80

Page 46: The Search for an Optimal Immunological Surrogate Endpoint ...faculty.washington.edu/peterg/Sanofi2018/Slides/GilbertEstimated... · Surrogate Endpoint FrameworksOptimal Surrogate

Surrogate Endpoint Frameworks Optimal Surrogate Framework Simulation Studies Application to Dengue Trials Discussion

Candidate Surrogate Outcomes (True UnknownParameters)

Any real-valued function (W ,A, S)→ ψ(W ,A,S) is acandidate surrogate, representing a measurement one cancollect by time τ and depending on the unknown trueobserved data distribution P0

Question: How to define a good surrogate in terms of P0?

Starting point: Only consider Sψ ≡ ψ(W ,A, S) that arevalid in the efficacy study, according to the Prentice definition:

VE = 0 ⇐⇒ Vaccine effect on the mean of Sψ = 0

24 / 80

Page 47: The Search for an Optimal Immunological Surrogate Endpoint ...faculty.washington.edu/peterg/Sanofi2018/Slides/GilbertEstimated... · Surrogate Endpoint FrameworksOptimal Surrogate

Surrogate Endpoint Frameworks Optimal Surrogate Framework Simulation Studies Application to Dengue Trials Discussion

Optimal Surrogate Outcome

Criterion for ranking valid surrogates and defining aP0-optimal surrogate:

Mean squared error MSE (ψ)Summarizes how close the outcome values Yi are to thesurrogate outcome values ψ(Wi ,Ai ,Si )

P0-optimal surrogate = the function ψ of (W ,A, S) thatminimizes MSE (ψ) subject to the Prentice definitionconstraint

25 / 80

Page 48: The Search for an Optimal Immunological Surrogate Endpoint ...faculty.washington.edu/peterg/Sanofi2018/Slides/GilbertEstimated... · Surrogate Endpoint FrameworksOptimal Surrogate

Surrogate Endpoint Frameworks Optimal Surrogate Framework Simulation Studies Application to Dengue Trials Discussion

Optimal Surrogate Outcome

Result 1

The minimizer of ψ → MSE (ψ) over all functions(W ,A,S)→ ψ(W ,A,S) that satisfy the Prentice definition is theconditional disease risk:

S0 = ψ0(W ,A,S) ≡ P0(Y = 1 |W ,A,S)

Advantageous Implication:

The vaccine effect on the optimal surrogate,

VE (S0) = 1− Mean of S0 Vaccine Group

Mean of S0 Placebo Group,

has the same scale of interpretation as

VE = 1− Mean of Y Vaccine Group

Mean of Y Placebo Group

26 / 80

Page 49: The Search for an Optimal Immunological Surrogate Endpoint ...faculty.washington.edu/peterg/Sanofi2018/Slides/GilbertEstimated... · Surrogate Endpoint FrameworksOptimal Surrogate

Surrogate Endpoint Frameworks Optimal Surrogate Framework Simulation Studies Application to Dengue Trials Discussion

Optimal Surrogate Outcome

Result 1

The minimizer of ψ → MSE (ψ) over all functions(W ,A,S)→ ψ(W ,A,S) that satisfy the Prentice definition is theconditional disease risk:

S0 = ψ0(W ,A,S) ≡ P0(Y = 1 |W ,A,S)

Advantageous Implication:

The vaccine effect on the optimal surrogate,

VE (S0) = 1− Mean of S0 Vaccine Group

Mean of S0 Placebo Group,

has the same scale of interpretation as

VE = 1− Mean of Y Vaccine Group

Mean of Y Placebo Group26 / 80

Page 50: The Search for an Optimal Immunological Surrogate Endpoint ...faculty.washington.edu/peterg/Sanofi2018/Slides/GilbertEstimated... · Surrogate Endpoint FrameworksOptimal Surrogate

Surrogate Endpoint Frameworks Optimal Surrogate Framework Simulation Studies Application to Dengue Trials Discussion

Estimation of the P0-optimal Surrogate

In practice, of course, the P0-optimal surrogateP0(Y = 1 |W ,A, S) is not available for use

It is estimated and the estimated regression functionP0(Y = 1 |W ,A,S) is used as the surrogate

i.e., individual i with covariates (Wi ,Ai ,Si ) has surrogateendpoint value P0(Yi = 1 | |Wi ,Ai ,Si )

27 / 80

Page 51: The Search for an Optimal Immunological Surrogate Endpoint ...faculty.washington.edu/peterg/Sanofi2018/Slides/GilbertEstimated... · Surrogate Endpoint FrameworksOptimal Surrogate

Surrogate Endpoint Frameworks Optimal Surrogate Framework Simulation Studies Application to Dengue Trials Discussion

Estimation of the P0-optimal Surrogate

In practice, of course, the P0-optimal surrogateP0(Y = 1 |W ,A, S) is not available for use

It is estimated and the estimated regression functionP0(Y = 1 |W ,A, S) is used as the surrogate

i.e., individual i with covariates (Wi ,Ai , Si ) has surrogateendpoint value P0(Yi = 1 | |Wi ,Ai ,Si )

27 / 80

Page 52: The Search for an Optimal Immunological Surrogate Endpoint ...faculty.washington.edu/peterg/Sanofi2018/Slides/GilbertEstimated... · Surrogate Endpoint FrameworksOptimal Surrogate

Surrogate Endpoint Frameworks Optimal Surrogate Framework Simulation Studies Application to Dengue Trials Discussion

Estimation of the P0-optimal Surrogate

Objective: Estimate the regression functionP0(Y = 1 | |W ,A,S) – a standard prediction problem

Challenge: A very large number of estimators are possible –How to achieve a best estimator?

i.e., how to optimally make the bias-variance tradeoff?Understanding bias/variance tradeoff in estimation

How big should h be?

Big h ñ more patients to borrow info from ñ low variance.

10 / 47

28 / 80

Page 53: The Search for an Optimal Immunological Surrogate Endpoint ...faculty.washington.edu/peterg/Sanofi2018/Slides/GilbertEstimated... · Surrogate Endpoint FrameworksOptimal Surrogate

Surrogate Endpoint Frameworks Optimal Surrogate Framework Simulation Studies Application to Dengue Trials Discussion

Estimation of the P0-optimal Surrogate

Objective: Estimate the regression functionP0(Y = 1 | |W ,A,S) – a standard prediction problem

Challenge: A very large number of estimators are possible –How to achieve a best estimator?

i.e., how to optimally make the bias-variance tradeoff?Understanding bias/variance tradeoff in estimation

How big should h be?

Big h ñ more patients to borrow info from ñ low variance.

10 / 47

28 / 80

Page 54: The Search for an Optimal Immunological Surrogate Endpoint ...faculty.washington.edu/peterg/Sanofi2018/Slides/GilbertEstimated... · Surrogate Endpoint FrameworksOptimal Surrogate

Surrogate Endpoint Frameworks Optimal Surrogate Framework Simulation Studies Application to Dengue Trials Discussion

How to Best Estimate P0(Y = 1 | |W ,A, S)?

Different regression methods tradeoff bias and variance indifferent ways

Nonparametric: Empirical moment, kernel regression, neuralnetworks, random forestsSemiparametric: Generalized additive models, partially linearadditive modelsParametric: Logistic regression, spline regression

For a given regression method, the tradeoff is governed bymodeling choices and/or tuning parameters

Logistic regression with two immune response biomakers(include an interaction term?)Uniform kernel estimator (large or small smoothingbandwidth?)Regression tree (maximum depth one versus thirty?)

The best bias/variance tradeoff depends on the (unknown)true regression function

29 / 80

Page 55: The Search for an Optimal Immunological Surrogate Endpoint ...faculty.washington.edu/peterg/Sanofi2018/Slides/GilbertEstimated... · Surrogate Endpoint FrameworksOptimal Surrogate

Surrogate Endpoint Frameworks Optimal Surrogate Framework Simulation Studies Application to Dengue Trials Discussion

How to Best Estimate P0(Y = 1 | |W ,A, S)?

Different regression methods tradeoff bias and variance indifferent ways

Nonparametric: Empirical moment, kernel regression, neuralnetworks, random forestsSemiparametric: Generalized additive models, partially linearadditive modelsParametric: Logistic regression, spline regression

For a given regression method, the tradeoff is governed bymodeling choices and/or tuning parameters

Logistic regression with two immune response biomakers(include an interaction term?)Uniform kernel estimator (large or small smoothingbandwidth?)Regression tree (maximum depth one versus thirty?)

The best bias/variance tradeoff depends on the (unknown)true regression function

29 / 80

Page 56: The Search for an Optimal Immunological Surrogate Endpoint ...faculty.washington.edu/peterg/Sanofi2018/Slides/GilbertEstimated... · Surrogate Endpoint FrameworksOptimal Surrogate

Surrogate Endpoint Frameworks Optimal Surrogate Framework Simulation Studies Application to Dengue Trials Discussion

Super-Learner Estimator of P0(Y = 1 | |W ,A, S)

1 Specify a large library of regression methods/estimators forP0(Y = 1 | |W ,A,S)

2 Use a fair prediction performance criterion to compare all theestimators

3 Select the best estimator by this criterion

called the Discrete Super Learner

4 Also select the best combination estimator that is the bestweighted average of all of the individual estimators

called the Super Learner

30 / 80

Page 57: The Search for an Optimal Immunological Surrogate Endpoint ...faculty.washington.edu/peterg/Sanofi2018/Slides/GilbertEstimated... · Surrogate Endpoint FrameworksOptimal Surrogate

Surrogate Endpoint Frameworks Optimal Surrogate Framework Simulation Studies Application to Dengue Trials Discussion

Fair Criterion: Cross-Validated Prediction Performance

Divide data into V sets of size ≈ nV (Here, V = 10)

Fold 1 = training sample T 1 + validation sample V1Training sample is used to fit (“train”) the regressions

Validation sample is used to estimate prediction performance(“validate”)

Several factors to consider when choosing V :

Large V = more data to fit regressions (helpful in small datasets or with high-dimensional covariates)

Small V = more data to evaluate prediction performance

Large V = greater computation time

31 / 80

Page 58: The Search for an Optimal Immunological Surrogate Endpoint ...faculty.washington.edu/peterg/Sanofi2018/Slides/GilbertEstimated... · Surrogate Endpoint FrameworksOptimal Surrogate

Surrogate Endpoint Frameworks Optimal Surrogate Framework Simulation Studies Application to Dengue Trials Discussion

Fair Criterion: Cross-Validated Prediction Performance

Divide data into V sets of size ≈ nV (Here, V = 10)

Fold 1 = training sample T 1 + validation sample V1Training sample is used to fit (“train”) the regressions

Validation sample is used to estimate prediction performance(“validate”)

Several factors to consider when choosing V :

Large V = more data to fit regressions (helpful in small datasets or with high-dimensional covariates)

Small V = more data to evaluate prediction performance

Large V = greater computation time

31 / 80

Page 59: The Search for an Optimal Immunological Surrogate Endpoint ...faculty.washington.edu/peterg/Sanofi2018/Slides/GilbertEstimated... · Surrogate Endpoint FrameworksOptimal Surrogate

Surrogate Endpoint Frameworks Optimal Surrogate Framework Simulation Studies Application to Dengue Trials Discussion

Fair Criterion: Cross-Validated Prediction PerformanceEstimation via cross-validated risk assessment

The validation set rotates until each set has been used as validation once.

9

1

2

3

4

5

6

7

8

10

9

1

2

3

4

5

6

7

8

10

9

1

2

3

4

5

6

7

8

10

9

1

2

3

4

5

6

7

8

10

9

1

2

3

4

5

6

7

8

10

9

1

2

3

4

5

6

7

8

10

9

1

2

3

4

5

6

7

8

10

9

1

2

3

4

5

6

7

8

10

9

1

2

3

4

5

6

7

8

10

9

1

2

3

4

5

6

7

8

10

Fold 1 Fold 2 Fold 3 Fold 4 Fold 5 Fold 6 Fold 7 Fold 8 Fold 9 Fold 10 24 / 4732 / 80

Page 60: The Search for an Optimal Immunological Surrogate Endpoint ...faculty.washington.edu/peterg/Sanofi2018/Slides/GilbertEstimated... · Surrogate Endpoint FrameworksOptimal Surrogate

Surrogate Endpoint Frameworks Optimal Surrogate Framework Simulation Studies Application to Dengue Trials Discussion

Measure Cross-Validated Prediction Performance byCross-Validated Risk

Risk = average loss of an estimator, where the loss scores howfar away the prediction of Y made from the estimatorP0(Y = 1|A,W ,S) is from the true Y

E.g., squared error loss (Yi − P0(Yi = 1|Wi ,Ai ,Si ))2

Cross-Validated Risk

1 Build the model from training set T 1; estimate risk onvalidation set V1

2 Build the model from training set T 2; estimate risk onvalidation set V2·

10 Build the model from training set T 10; estimate risk onvalidation set V10

Cross-validated risk = average of the 10 validation set risks

33 / 80

Page 61: The Search for an Optimal Immunological Surrogate Endpoint ...faculty.washington.edu/peterg/Sanofi2018/Slides/GilbertEstimated... · Surrogate Endpoint FrameworksOptimal Surrogate

Surrogate Endpoint Frameworks Optimal Surrogate Framework Simulation Studies Application to Dengue Trials Discussion

Measure Cross-Validated Prediction Performance byCross-Validated Risk

Risk = average loss of an estimator, where the loss scores howfar away the prediction of Y made from the estimatorP0(Y = 1|A,W ,S) is from the true Y

E.g., squared error loss (Yi − P0(Yi = 1|Wi ,Ai ,Si ))2

Cross-Validated Risk

1 Build the model from training set T 1; estimate risk onvalidation set V1

2 Build the model from training set T 2; estimate risk onvalidation set V2·

10 Build the model from training set T 10; estimate risk onvalidation set V10

Cross-validated risk = average of the 10 validation set risks33 / 80

Page 62: The Search for an Optimal Immunological Surrogate Endpoint ...faculty.washington.edu/peterg/Sanofi2018/Slides/GilbertEstimated... · Surrogate Endpoint FrameworksOptimal Surrogate

Surrogate Endpoint Frameworks Optimal Surrogate Framework Simulation Studies Application to Dengue Trials Discussion

Super Learner Based on Cross-Validated Risk

Discrete Super Learner is the estimator P0(Y = 1|W ,A, S)with the smallest cross-validated risk

Super Learner is the weighted average of all of the estimatorsP0(Y = 1|W ,A, S) with the smallest cross-validated risk

Idea originated with “model stacking” of Wolpert (1992) andBreiman (1996)Idea generalized and re-branded as “super learning” (van derLaan, Polley, and Hubbard, 2007)

34 / 80

Page 63: The Search for an Optimal Immunological Surrogate Endpoint ...faculty.washington.edu/peterg/Sanofi2018/Slides/GilbertEstimated... · Surrogate Endpoint FrameworksOptimal Surrogate

Surrogate Endpoint Frameworks Optimal Surrogate Framework Simulation Studies Application to Dengue Trials Discussion

Strong Practical Performance of Super-Learner∗Super learning and oracle inequalities

44 / 47

∗E.g., van der Laan et al. (2007), van der Laan and Rose (2011),

Pirracchioet al. (2015), Petersen et al. (2015), Acion et al. (2017)35 / 80

Page 64: The Search for an Optimal Immunological Surrogate Endpoint ...faculty.washington.edu/peterg/Sanofi2018/Slides/GilbertEstimated... · Surrogate Endpoint FrameworksOptimal Surrogate

Surrogate Endpoint Frameworks Optimal Surrogate Framework Simulation Studies Application to Dengue Trials Discussion

Super-Learner with Cross-Validated Classification AccuracyMetrics∗: A Framework for Comparing Estimated OptimalSurrogates and Seeking Parsimonious Surrogates

Example 1 Example 2

0.6 0.7 0.8 0.9 0.6 0.7 0.8 0.9

Baseline Variables (W)

Marker S1

Marker S1 + Markers S2−S50

W + Marker S1

W + Marker S1 + Markers S2−S50

Cross−Validated Area Under the ROC Curves with 95% CIs

Inpu

t Var

iabl

e S

ets

∗ Van der Laan, Hubbard, and Pajouh (2013)36 / 80

Page 65: The Search for an Optimal Immunological Surrogate Endpoint ...faculty.washington.edu/peterg/Sanofi2018/Slides/GilbertEstimated... · Surrogate Endpoint FrameworksOptimal Surrogate

Surrogate Endpoint Frameworks Optimal Surrogate Framework Simulation Studies Application to Dengue Trials Discussion

Advantageous Properties of Super Learner

Oracle Property: It has predictive performance risk very closeto the oracle estimator that uses the true (unknown) bestestimator

Flexibility: The number of estimators is allowed to be verylarge – and including a large number of estimators in thelibrary of learners aids achieving the oracle property

Any given regression method can be used to constructmultiple different estimators, e.g.:

Random forest with different tuning parametersGeneralized additive models with different knots and degreesLogistic regression with interactions and stepwise selection

37 / 80

Page 66: The Search for an Optimal Immunological Surrogate Endpoint ...faculty.washington.edu/peterg/Sanofi2018/Slides/GilbertEstimated... · Surrogate Endpoint FrameworksOptimal Surrogate

Surrogate Endpoint Frameworks Optimal Surrogate Framework Simulation Studies Application to Dengue Trials Discussion

Advantageous Properties of Super Learner

Oracle Property: It has predictive performance risk very closeto the oracle estimator that uses the true (unknown) bestestimator

Flexibility: The number of estimators is allowed to be verylarge – and including a large number of estimators in thelibrary of learners aids achieving the oracle property

Any given regression method can be used to constructmultiple different estimators, e.g.:

Random forest with different tuning parametersGeneralized additive models with different knots and degreesLogistic regression with interactions and stepwise selection

37 / 80

Page 67: The Search for an Optimal Immunological Surrogate Endpoint ...faculty.washington.edu/peterg/Sanofi2018/Slides/GilbertEstimated... · Surrogate Endpoint FrameworksOptimal Surrogate

Surrogate Endpoint Frameworks Optimal Surrogate Framework Simulation Studies Application to Dengue Trials Discussion

Advantageous Properties of Super Learner

Oracle Property: It has predictive performance risk very closeto the oracle estimator that uses the true (unknown) bestestimator

Flexibility: The number of estimators is allowed to be verylarge – and including a large number of estimators in thelibrary of learners aids achieving the oracle property

Any given regression method can be used to constructmultiple different estimators, e.g.:

Random forest with different tuning parametersGeneralized additive models with different knots and degreesLogistic regression with interactions and stepwise selection

37 / 80

Page 68: The Search for an Optimal Immunological Surrogate Endpoint ...faculty.washington.edu/peterg/Sanofi2018/Slides/GilbertEstimated... · Surrogate Endpoint FrameworksOptimal Surrogate

Surrogate Endpoint Frameworks Optimal Surrogate Framework Simulation Studies Application to Dengue Trials Discussion

Advantageous Properties of Super Learner

Traditional practice tries several models and checks model fitto select a model

This exploration practice without pre-specification invalidatesinferences

In contrast, Super Learner is pre-specified

Eliminates the need to put all our eggs in a single estimationbasketInclude in the library any model choice that could result frommodel checkingOracle property ensures that Super Learner is good atchoosing (approximately) the correct one

A major scientific activity is selection of the library ofestimators

38 / 80

Page 69: The Search for an Optimal Immunological Surrogate Endpoint ...faculty.washington.edu/peterg/Sanofi2018/Slides/GilbertEstimated... · Surrogate Endpoint FrameworksOptimal Surrogate

Surrogate Endpoint Frameworks Optimal Surrogate Framework Simulation Studies Application to Dengue Trials Discussion

Advantageous Properties of Super Learner

Traditional practice tries several models and checks model fitto select a model

This exploration practice without pre-specification invalidatesinferences

In contrast, Super Learner is pre-specified

Eliminates the need to put all our eggs in a single estimationbasketInclude in the library any model choice that could result frommodel checkingOracle property ensures that Super Learner is good atchoosing (approximately) the correct one

A major scientific activity is selection of the library ofestimators

38 / 80

Page 70: The Search for an Optimal Immunological Surrogate Endpoint ...faculty.washington.edu/peterg/Sanofi2018/Slides/GilbertEstimated... · Surrogate Endpoint FrameworksOptimal Surrogate

Surrogate Endpoint Frameworks Optimal Surrogate Framework Simulation Studies Application to Dengue Trials Discussion

Advantageous Properties of Super Learner

Traditional practice tries several models and checks model fitto select a model

This exploration practice without pre-specification invalidatesinferences

In contrast, Super Learner is pre-specified

Eliminates the need to put all our eggs in a single estimationbasketInclude in the library any model choice that could result frommodel checkingOracle property ensures that Super Learner is good atchoosing (approximately) the correct one

A major scientific activity is selection of the library ofestimators

38 / 80

Page 71: The Search for an Optimal Immunological Surrogate Endpoint ...faculty.washington.edu/peterg/Sanofi2018/Slides/GilbertEstimated... · Surrogate Endpoint FrameworksOptimal Surrogate

Surrogate Endpoint Frameworks Optimal Surrogate Framework Simulation Studies Application to Dengue Trials Discussion

Super Learner-Based Estimated Optimal Surrogate

Estimated optimal surrogate (EOS): ˆS0 = P0(Y |W ,A, S)

Vaccine effect on the EOS ˆS0:

VE (ˆS0) = 1− Mean of ˆS0 Vaccine Group

Mean of ˆS0 Placebo Group

Vaccine effect on the disease endpoint Y :

VE0 = 1− Mean of Y Vaccine Group

Mean of Y Placebo Group

Price, Gilbert, and van der Laan (2018) showed how to

estimate VE (ˆS0) with a confidence intervalThey showed that a TMLE-adjusted Super Learner estimatoris an asymptotically efficient estimator of VE

A desirable property of a best surrogate built from anefficacy trial

39 / 80

Page 72: The Search for an Optimal Immunological Surrogate Endpoint ...faculty.washington.edu/peterg/Sanofi2018/Slides/GilbertEstimated... · Surrogate Endpoint FrameworksOptimal Surrogate

Surrogate Endpoint Frameworks Optimal Surrogate Framework Simulation Studies Application to Dengue Trials Discussion

Special Application of the EOS if Prentice’s (1989) “FullMediation Condition” Holds

The key Prentice criterion for a surrogate endpoint to be validis: Within each subgroup defined by (W , S), disease riskis the same in the vaccine and placebo groups

P(Y = 1|W = w ,A = 1, S = s) = P(Y = 1|W = w ,A = 0, S = s)

Typically fails, but may hold if the surrogate is tightly linkedto a mechanism of protection, that operates the same forvaccine immunity vs. natural immunity

40 / 80

Page 73: The Search for an Optimal Immunological Surrogate Endpoint ...faculty.washington.edu/peterg/Sanofi2018/Slides/GilbertEstimated... · Surrogate Endpoint FrameworksOptimal Surrogate

Surrogate Endpoint Frameworks Optimal Surrogate Framework Simulation Studies Application to Dengue Trials Discussion

Example: Dunning et al. (2016, Clin Vacc Immun)

VE trial of Sanofi’sInactivatedInfluenza VaccineHigh vs. StandardDose in ≥ 65year-olds

Correlates ofprotection analysisof antibodies (HAI,NAI, NT –neutralization test)

Article concludedthat combiningassays improvedsurrogate quality

Protection curves for the A/Victoria/361/2011 HAI assay using the circulating virus against A/H3N2 illness by three laboratory-confirmed influenza (LCI) case definitions (defn.), showing

titers for 50% and 80% protection, with 95% CIs.

Andrew J. Dunning et al. Clin. Vaccine Immunol. 2016; doi:10.1128/CVI.00604-15

Could repeat with the EOS P0(W , S)on the x-axis using HAI, NAI, NT

41 / 80

Page 74: The Search for an Optimal Immunological Surrogate Endpoint ...faculty.washington.edu/peterg/Sanofi2018/Slides/GilbertEstimated... · Surrogate Endpoint FrameworksOptimal Surrogate

Surrogate Endpoint Frameworks Optimal Surrogate Framework Simulation Studies Application to Dengue Trials Discussion

Goal 2: Bridging/Transportability

Goal 2: Use the P0-estimated optimal surrogate built from aprevious efficacy trial(s) as the primary study endpoint in afuture clinical trial, for inference on VE without measuring Y

Surrogate endpoint S0i = Super Learner P0(Yi = 1 |Wi ,Ai ,Si )

This bridging problem is hard given the implicit necessity ofextrapolating beyond the empirical data

We give (strong) conditions under which this bridginginference may be done in a valid way

42 / 80

Page 75: The Search for an Optimal Immunological Surrogate Endpoint ...faculty.washington.edu/peterg/Sanofi2018/Slides/GilbertEstimated... · Surrogate Endpoint FrameworksOptimal Surrogate

Surrogate Endpoint Frameworks Optimal Surrogate Framework Simulation Studies Application to Dengue Trials Discussion

Goal 2: Bridging/Transportability

Goal 2: Use the P0-estimated optimal surrogate built from aprevious efficacy trial(s) as the primary study endpoint in afuture clinical trial, for inference on VE without measuring Y

Surrogate endpoint S0i = Super Learner P0(Yi = 1 |Wi ,Ai ,Si )

This bridging problem is hard given the implicit necessity ofextrapolating beyond the empirical data

We give (strong) conditions under which this bridginginference may be done in a valid way

42 / 80

Page 76: The Search for an Optimal Immunological Surrogate Endpoint ...faculty.washington.edu/peterg/Sanofi2018/Slides/GilbertEstimated... · Surrogate Endpoint FrameworksOptimal Surrogate

Surrogate Endpoint Frameworks Optimal Surrogate Framework Simulation Studies Application to Dengue Trials Discussion

Assumptions Under which the P0-Optimal Surrogate Canbe Used for Valid Estimation of VE ∗ in the New Study

Theorem 2 from Price et al. (2018)

Consider a new randomized study with collected data(W ∗

i ,A∗i , S

∗i ), i = 1, · · · , n∗

The Following Assumptions Guarantee Correct Bridging:

Equal Conditional Disease Risk: Within each subgroupdefined by (W ∗,A∗,S∗), disease risk is the same in theoriginal and new studies

Contained Support: All of the subgroups defined by(W ∗,A∗,S∗) are represented in the original study

Positivity: All subgroups defined by W ∗ are represented inboth the vaccine and placebo groups A∗ = 1 and A∗ = 0

43 / 80

Page 77: The Search for an Optimal Immunological Surrogate Endpoint ...faculty.washington.edu/peterg/Sanofi2018/Slides/GilbertEstimated... · Surrogate Endpoint FrameworksOptimal Surrogate

Surrogate Endpoint Frameworks Optimal Surrogate Framework Simulation Studies Application to Dengue Trials Discussion

1 Surrogate Endpoint Frameworks

2 Optimal Surrogate Framework

3 Simulation Studies

4 Application to Dengue Trials

5 Discussion

44 / 80

Page 78: The Search for an Optimal Immunological Surrogate Endpoint ...faculty.washington.edu/peterg/Sanofi2018/Slides/GilbertEstimated... · Surrogate Endpoint FrameworksOptimal Surrogate

Surrogate Endpoint Frameworks Optimal Surrogate Framework Simulation Studies Application to Dengue Trials Discussion

Two Simulation Studies

Objective of First Study: Simple illustration that theestimated optimal surrogate will always provide unbiasedestimation of VE0 = P0(Y1 − Y0) in the original trial, for anydistribution of (W ,A,S ,Y )

Objective of Second Study: Illustrate how well theestimated optimal surrogate built from one trial works forinference on VE ∗ = EP(Y ∗

1 − Y ∗0 ) in a second trial, when

Equal Conditional Disease Risk fails

45 / 80

Page 79: The Search for an Optimal Immunological Surrogate Endpoint ...faculty.washington.edu/peterg/Sanofi2018/Slides/GilbertEstimated... · Surrogate Endpoint FrameworksOptimal Surrogate

Surrogate Endpoint Frameworks Optimal Surrogate Framework Simulation Studies Application to Dengue Trials Discussion

Data Demonstrating the Surrogate Paradox

Building upon an example published by VanderWeele (2014,Biometrics)

Continuous outcome Y

Treatment A ∈ {0, 1}10 candidate surrogates S i (S i ∈ {0, 1, 2}, i = 1 . . . 10)

P(S i1 = 0, S i

0 = 0) = P(S i1 = 1, S i

0 = 1) = P(S i1 = 2, S i

0 = 2) = 0.1,

P(S i1 = 1, S i

0 = 0) = 0.5,

P(S i1 = 1, S i

0 = 2) = 0.2

Y =∑3

i=1 [0.1 ∗ i ∗ IS i=1 + IS i=2] + εY , εY ∼ N(0, 0.12)

46 / 80

Page 80: The Search for an Optimal Immunological Surrogate Endpoint ...faculty.washington.edu/peterg/Sanofi2018/Slides/GilbertEstimated... · Surrogate Endpoint FrameworksOptimal Surrogate

Surrogate Endpoint Frameworks Optimal Surrogate Framework Simulation Studies Application to Dengue Trials Discussion

The Surrogate Paradox Occurs

1 S → YPOSITIVE relationship between surrogates and outcomeY =

∑3i=1 [0.1 ∗ i ∗ IS i=1 + IS i=2] + εY , εY ∼ N(0, 0.12);

2 A→ SPOSITIVE treatment effect on surrogatesE [S i

1 − S i0] = 0.3;

3 A→ YNEGATIVE overall treatment effectE [Y1 − Y0] = −0.18

47 / 80

Page 81: The Search for an Optimal Immunological Surrogate Endpoint ...faculty.washington.edu/peterg/Sanofi2018/Slides/GilbertEstimated... · Surrogate Endpoint FrameworksOptimal Surrogate

Surrogate Endpoint Frameworks Optimal Surrogate Framework Simulation Studies Application to Dengue Trials Discussion

Simulation 1: Compare the TMLE-SL Estimator of VE0 toa Standard Estimator

Standard estimator of VE0: Simple regression estimator afterselection of the surrogate based on the Proportion of theTreatment Effect Captured (PCS) by the candidate Surrogate∗

For each S i , estimate PCS nonparametrically

PCS =CP2

CP2 + NCP2

(true PCS = 0.87, 0.2, 0.002 for i = 1, 2, 3; PCS = 0 for i = 4, . . . , 10)

Select “best surrogate”: SPCSopt = S i with the greatest PCS

Estimate VE0 by the difference (a = 1 minus a = 0) in

average predicted Y’s P(Yi = 1 | SPCSopti ,Ai = a)

∗Kobayashi F, Kuroki M (2014) A new proportion measure of the

treatment effect captured by candidate surrogate endpoints, Stat Med48 / 80

Page 82: The Search for an Optimal Immunological Surrogate Endpoint ...faculty.washington.edu/peterg/Sanofi2018/Slides/GilbertEstimated... · Surrogate Endpoint FrameworksOptimal Surrogate

Surrogate Endpoint Frameworks Optimal Surrogate Framework Simulation Studies Application to Dengue Trials Discussion

Simulation 1: Estimation under the Surrogate Paradox(n = 2000 Subjects; 200 Simulated Data Sets)

Surrogate Paradox

Occurs for 95% ofPCS∗ methodestimates: VE 0 > 0(vs. truthVE0 = −0.18)Does not occur withTMLE-SL method:VE 0 < 0

∗Proportion of treatmenteffect captured (PCS)(Kobayashi and Kuroki,2014, Stat Med)

●●

●●●

●●●

●●

●●

●●● ●

●●

●●

● ●

●●

● ●

●● ●●

● ●●●

● ● ●●

●●●●

●●

●●

●●

●●

● ●

●● ●● ●●

●● ●

●●

●●●

●●●

●●●

● ●

●●

●●

●● ●

●●

●●

●●

●●

●●

●●

●●

●●

●●●

●●● ●

●●

●●

●●

●●

●●●

●●

−0.20

−0.15

−0.10

−0.05

0.00

−0.20 −0.15 −0.10 −0.05 0.00 Estimated Treatment Effect on Y Based on Y

Est

imat

ed T

reat

men

t Effe

ct o

n Y

Bas

ed o

n ea

ch S

urro

gate

Estimate●

a) Concordance of Estimates (Study D1)

●●●

●●

●● ●

●●

●●

●●

●●

●● ●●

●●

●●●

●●● ● ●

●●●

●●●●

●●

● ●

●●●

● ●●

●●

●● ●

● ● ●●

●●

●●

●●

● ●

●●●

●●●●

● ●

● ●

●●

●●

●●

●●● ●

●●

●●

●●

●● ● ●

●●●

● ●●

●●

●●

●● ●

●●

−0.20

−0.15

−0.10

−0.05

0.00

−0.20 −0.15 −0.10 −0.05 0.00Estimated Treatment Effect on Y* Based on Y* (θ~n*

TMLE)

Est

imat

ed T

reat

men

t Effe

ct o

n Y

* Bas

ed o

n ea

ch S

urro

gate

Estimate● θn

PCSopt(P)θψn

TMLE(P)

b) Concordance of Estimates (Study D2)

49 / 80

pgilbert
Typewritten Text
PCS
pgilbert
Typewritten Text
pgilbert
Typewritten Text
pgilbert
Typewritten Text
pgilbert
Typewritten Text
pgilbert
Typewritten Text
pgilbert
Typewritten Text
pgilbert
Typewritten Text
TMLE-SL
pgilbert
Typewritten Text
pgilbert
Typewritten Text
pgilbert
Typewritten Text
pgilbert
Typewritten Text
pgilbert
Typewritten Text
pgilbert
Typewritten Text
pgilbert
Typewritten Text
pgilbert
Typewritten Text
Page 83: The Search for an Optimal Immunological Surrogate Endpoint ...faculty.washington.edu/peterg/Sanofi2018/Slides/GilbertEstimated... · Surrogate Endpoint FrameworksOptimal Surrogate

Surrogate Endpoint Frameworks Optimal Surrogate Framework Simulation Studies Application to Dengue Trials Discussion

Simulation 2: Transportability When Equal ConditionalDisease Risks Fails

Both the PCS and theTMLE-SL methods arebiased

Surrogate Paradox

Occurs for 95% of PCSmethod estimates:VE∗> 0

(vs. truthVE∗ = −0.10)Does not occur withthe TMLE-SL method:VE∗< 0

●●

●●●

●●●

●●

●●

●●● ●

●●

●●

● ●

●●

● ●

●● ●●

● ●●●

● ● ●●

●●●●

●●

●●

●●

●●

● ●

●● ●● ●●

●● ●

●●

●●●

●●●

●●●

● ●

●●

●●

●● ●

●●

●●

●●

●●

●●

●●

●●

●●

●●●

●●● ●

●●

●●

●●

●●

●●●

●●

−0.20

−0.15

−0.10

−0.05

0.00

−0.20 −0.15 −0.10 −0.05 0.00Estimated Treatment Effect on Y Based on Y (θ~n

TMLE)

Est

imat

ed T

reat

men

t Effe

ct o

n Y

Bas

ed o

n ea

ch S

urro

gate

Estimate● θn

PCSopt

θnTMLE

a) Concordance of Estimates (Study D1)

●●●

●●

●● ●

●●

●●

●●

●●

●● ●●

●●

●●●

●●● ● ●

●●●

●●●●

●●

● ●

●●●

● ●●

●●

●● ●

● ● ●●

●●

●●

●●

● ●

●●●

●●●●

● ●

● ●

●●

●●

●●

●●● ●

●●

●●

●●

●● ● ●

●●●

● ●●

●●

●●

●● ●

●●

−0.20

−0.15

−0.10

−0.05

0.00

−0.20 −0.15 −0.10 −0.05 0.00 Estimated Treatment Effect on Y* Based on Y*

Est

imat

ed T

reat

men

t Effe

ct o

n Y

* Bas

ed o

n ea

ch S

urro

gate

Estimate●

b) Concordance of Estimates (Study D2)

50 / 80

pgilbert
Typewritten Text
PCS
pgilbert
Typewritten Text
TMLE-SL
Page 84: The Search for an Optimal Immunological Surrogate Endpoint ...faculty.washington.edu/peterg/Sanofi2018/Slides/GilbertEstimated... · Surrogate Endpoint FrameworksOptimal Surrogate

Surrogate Endpoint Frameworks Optimal Surrogate Framework Simulation Studies Application to Dengue Trials Discussion

Conclusion from Simulation 2

Demonstrates that the Equal Conditional Disease Risksassumption is necessary for valid inference about VE ∗ in anew setting

When Equal Conditional Disease Risks is majorly violated, theestimated optimal surrogate can still preserve some accuracyin bridging the clinical treatment effect to a new setting

51 / 80

Page 85: The Search for an Optimal Immunological Surrogate Endpoint ...faculty.washington.edu/peterg/Sanofi2018/Slides/GilbertEstimated... · Surrogate Endpoint FrameworksOptimal Surrogate

Surrogate Endpoint Frameworks Optimal Surrogate Framework Simulation Studies Application to Dengue Trials Discussion

1 Surrogate Endpoint Frameworks

2 Optimal Surrogate Framework

3 Simulation Studies

4 Application to Dengue Trials

5 Discussion

52 / 80

Page 86: The Search for an Optimal Immunological Surrogate Endpoint ...faculty.washington.edu/peterg/Sanofi2018/Slides/GilbertEstimated... · Surrogate Endpoint FrameworksOptimal Surrogate

Surrogate Endpoint Frameworks Optimal Surrogate Framework Simulation Studies Application to Dengue Trials Discussion

Dengue Phase 3 Trial Example

Two randomized, double-blinded, placebo-controlled,multicenter, Phase 3 trials of a recombinant, live, attenuated,tetravalent (4 serotypes) dengue vaccine (CYD-TDV)

CYD14: Asia-Pacific region, 2–14 year-olds (Capeding et al.,2014, The Lancet)CYD15: Latin America, 9–16 year-olds (Villar et al., 2015,NEJM)

Trial Designs

2:1 randomization to vaccine:placebo

Immunizations at months 0, 6, 12

Primary follow-up from Month 13 to Month 25 (active phaseof follow-up)

Primary endpoint: Symptomatic, virologically confirmeddengue (VCD)

53 / 80

Page 87: The Search for an Optimal Immunological Surrogate Endpoint ...faculty.washington.edu/peterg/Sanofi2018/Slides/GilbertEstimated... · Surrogate Endpoint FrameworksOptimal Surrogate

Surrogate Endpoint Frameworks Optimal Surrogate Framework Simulation Studies Application to Dengue Trials Discussion

Results on Vaccine Efficacy to Prevent VCD from Month13 to 25 (Proportional Hazards Models)

CYD14: VE = 56.5% (95% CI 43.8–66.4) CYD15: VE = 64.7% (95% CI 58.7–69.8)

N = 10, 275, n = 244 endpoints Y = 1 N∗ = 20, 869, n∗ = 415 endpoints Y∗ = 1

CYD15 Trial (Villar et al., 2015, NEJM)CYD14 Trial (Capeding et al., 2014, The Lancet)

54 / 80

Page 88: The Search for an Optimal Immunological Surrogate Endpoint ...faculty.washington.edu/peterg/Sanofi2018/Slides/GilbertEstimated... · Surrogate Endpoint FrameworksOptimal Surrogate

Surrogate Endpoint Frameworks Optimal Surrogate Framework Simulation Studies Application to Dengue Trials Discussion

Correlates of Risk and Correlates of VE Study in CYD14and CYD15 (Moodie et al., 2018, JID)

The Journal of Infectious Diseases

742 • JID 2018:217 (1 March) • Moodie et al

Neutralizing Antibody Correlates Analysis of Tetravalent Dengue Vaccine Efficacy Trials in Asia and Latin AmericaZoe Moodie,1 Michal Juraska,1 Ying Huang,1,2 Yingying Zhuang,2 Youyi Fong,1,2 Lindsay N. Carpp,1 Steven G. Self,1,2 Laurent Chambonneau,3 Robert Small,4 Nicholas Jackson,5 Fernando Noriega,4 and Peter B. Gilbert1,2

1Vaccine and Infectious Disease Division, Fred Hutchinson Cancer Research Center, Seattle, Washington; 2Department of Biostatistics, University of Washington, Seattle; 3Sanofi Pasteur, Marcy-L’Etoile, France; 4Sanofi Pasteur, Swiftwater, Pennsylvania; 5Sanofi Pasteur, Lyon, France

Background. In the CYD14 and CYD15 Phase 3 trials of the CYD-TDV dengue vaccine, estimated vaccine efficacy (VE) against symptomatic, virologically confirmed dengue (VCD) occurring between months 13 and 25 was 56.5% and 60.8%, respectively.

Methods. Neutralizing antibody titers to the 4 dengue serotypes in the CYD-TDV vaccine insert were measured at month 13 in a randomly sampled immunogenicity subcohort and in all VCD cases through month 25 (2848 vaccine, 1574 placebo) and studied for their association with VCD and with the level of VE to prevent VCD.

Results. For each trial and serotype, vaccinees with higher month 13 titer to the serotype had significantly lower risk of VCD with that serotype (hazard ratios, 0.19–0.43 per 10-fold increase). Moreover, for each trial, vaccinees with higher month 13 average titer to the 4 serotypes had significantly higher VE against VCD of any serotype (P < .001).

Conclusions. Neutralizing antibody titers postdose 3 correlate with CYD-TDV VE to prevent dengue. High titers associate with high VE for all serotypes, baseline serostatus groups, age groups, and both trials. However, lowest titers do not fully correspond to zero VE, indicating that other factors influence VE.

Keywords: case cohort; immune correlate of protection; neutralizing antibodies; surrogate endpoint; vaccine efficacy trial.

Although a satisfactory immune correlate of protection (CoP)—a biomarker measuring an immune response to vaccination that is strongly statistically associated with vaccine efficacy (VE) [1]—has not been established, a leading hypothesis is that a dengue vaccine must elicit functional neutralizing antibodies against all 4 serotypes to achieve high efficacy to prevent infection and dis-ease. Indeed, neutralizing antibodies are an established CoP (not necessarily mechanistic) for related flavivirus vaccines [2]. The ideal dengue vaccine would protect individuals seronegative to dengue, as well as individuals partially immune to dengue or to nondengue flaviviruses; these categories may have different CoPs.

Some studies have identified neutralizing antibody titers associated with protection from polymerase chain reac-tion-confirmed dengue infection [3] or symptomatic, virologi-cally confirmed dengue (VCD) [4, 5], but others have suggested that the humoral response alone is not reliably predictive [6–8].

Moreover, CD4+ and CD8+ T cells contribute to protection against disease-accompanying infections with heterotypic den-gue viruses [9–13], highlighting the complexity of dengue and the need to identify reliable CoPs.

CYD-TDV (Dengvaxia, Sanofi Pasteur) is a licensed recombi-nant, live, attenuated tetravalent dengue vaccine [14]. In 2 Phase 3 trials, CYD14 in 2- to 14-year-olds in Asia (NCT01373281) and CYD15 in 9- to 16-year-olds in Latin America (NCT01374516), estimated VE of CYD-TDV was 56.5% and 60.8% for the pre-vention of symptomatic, VCD of any serotype (DENV-Any) between 28 days after the third injection at 12 months through 25  months [15, 16]. Vaccine efficacy significantly varied by serotype, estimated at 50.0% (50.3%), 35.0% (42.3%), 78.4% (74.0%), and 75.3% (77.7%) against DENV-1, -2, -3, and -4 in CYD14 (CYD15) [15, 16]. However, the proportion of vaccine recipients with month 13 50% plaque reduction neutralization test (PRNT50) responses [17] above the lower limit of quanti-tation (LLOQ) (93.8%, 98.5%, 96.7%, and 96.8% in CYD14; 94.9%, 98.5%, 98.4%, and 98.1% in CYD15 for DENV-1, -2, -3, and -4) far exceeded the VE estimates [15, 16]. This discordance between VE rates and neutralization response rates, also seen in [7], indicated that PRNT50 neutralization response is not a com-pletely valid CoP. However, this aggregate data analysis provides less information about CoPs than an analysis incorporating individual-level titer data. Accordingly, a case-cohort study was conducted to investigate the association of month 13 neutraliz-ing antibody titers with dengue occurrence through month 25 and with the level of VE to prevent dengue, for each trial and

M A J O R A R T I C L E

© The Author(s) 2017. Published by Oxford University Press for the Infectious Diseases Society of America. All rights reserved. For permissions, e-mail: [email protected]: 10.1093/infdis/jix609

Received 17 May 2017; editorial decision 20 November 2017; accepted 27 November 2017; published online November 29, 2017.

Presented in part: Dengue Research Consortium Meeting, Sanofi Pasteur VaxDesign, Orlando, April 2017; P. B. Gilbert at the Harvard T.H. Chan School of Public Health Department of Immunology and Infectious Diseases, Boston, MA, April 2017; 38th Annual Conference of the International Society for Clinical Biostatistics, Vigo, Spain, July 2017; and 2017 Annual Meeting of the American Society of Tropical Medicine and Hygiene, Baltimore, MD, November 2017.

Correspondence: P. B. Gilbert, PhD, Vaccine and Infectious Disease Division, Fred Hutchinson Cancer Research Center, PO Box 19024, 1100 Fairview Ave North, Seattle, WA 98109 ([email protected]).

STANDARD

The Journal of Infectious Diseases® 2018;217:742–53

1

217

March

Downloaded from https://academic.oup.com/jid/article-abstract/217/5/742/4675250by Fred Hutchinson Cancer Research Center - Arnold Library useron 11 August 2018

55 / 80

Page 89: The Search for an Optimal Immunological Surrogate Endpoint ...faculty.washington.edu/peterg/Sanofi2018/Slides/GilbertEstimated... · Surrogate Endpoint FrameworksOptimal Surrogate

Surrogate Endpoint Frameworks Optimal Surrogate Framework Simulation Studies Application to Dengue Trials Discussion

Correlates of Risk and Correlates of VE Study in CYD14and CYD15 (Moodie et al., 2018, JID)

Month 13 PRNT50 and Microneutralization V2 neutralizationlevels measured from a case-cohort sample

Cases: All symptomatic VCD cases between Month 13 and25 (n=244 CYD14; n=415 CYD15)

Controls: All in the immunogenicity subset free of the VCDendpoint at Month 25 (n=1879 CYD14; n=1884 CYD15)

Cases: VCD endpoint

Controls: No VCD endpoint

Visit Month 0 6 12 13 18 25

× ×

× ×

0

56 / 80

Page 90: The Search for an Optimal Immunological Surrogate Endpoint ...faculty.washington.edu/peterg/Sanofi2018/Slides/GilbertEstimated... · Surrogate Endpoint FrameworksOptimal Surrogate

Surrogate Endpoint Frameworks Optimal Surrogate Framework Simulation Studies Application to Dengue Trials Discussion

Application of the Estimated Optimal Surrogate Approachto CYD14 and CYD15

Example in Price et al. (2018)

Treat CYD14 as the current trial; CYD15 as the future trial

Notation and Variables

W = Baseline covariates: age, sex, country-specific fractionsof VCD endpoints of each specific serotype

A = Vaccination status (1=vaccine; 0=placebo)

S = Month 13 PRNT50 and Microneutralization Version 2neutralization titers to the 4 vaccine strains (serotypes 1–4),average, min, max

Y = Disease outcome (1=VCD endpoint between Month 13and 25; 0 = no VCD endpoint by Month 25)

57 / 80

Page 91: The Search for an Optimal Immunological Surrogate Endpoint ...faculty.washington.edu/peterg/Sanofi2018/Slides/GilbertEstimated... · Surrogate Endpoint FrameworksOptimal Surrogate

Surrogate Endpoint Frameworks Optimal Surrogate Framework Simulation Studies Application to Dengue Trials Discussion

Month 13 PRNT50 Titer Data S by Levels of W (Sex,Age) and A (Vaccine, Placebo): CYD14

Vaccine, Male, Age: 2−8 Vaccine, Male, Age: 9−11 Vaccine, Male, Age: 12−14

Vaccine, Female, Age: 2−8 Vaccine, Female, Age: 9−11 Vaccine, Female, Age: 12−14

Placebo, Male, Age: 2−8 Placebo, Male, Age: 9−11 Placebo, Male, Age: 12−14

Placebo, Female, Age: 2−8 Placebo, Female, Age: 9−11 Placebo, Female, Age: 12−14

Type 1 Type 2 Type 3 Type 4 Type 1 Type 2 Type 3 Type 4 Type 1 Type 2 Type 3 Type 4

0

2

4

0

2

4

0

2

4

0

2

4

Serotypes

Mon

th 1

3 Lo

g 10

PR

NT

50 N

eutr

aliz

atio

n T

iter

CYD14

58 / 80

Page 92: The Search for an Optimal Immunological Surrogate Endpoint ...faculty.washington.edu/peterg/Sanofi2018/Slides/GilbertEstimated... · Surrogate Endpoint FrameworksOptimal Surrogate

Surrogate Endpoint Frameworks Optimal Surrogate Framework Simulation Studies Application to Dengue Trials Discussion

Month 13 PRNT50 Average Titers∗: CYD14 and CYD15Assess Month 13 Average nAb Titer* as a Fixed-Time CoR and CoP (Moodie et al., 2018, JID)

0

CYD14 Case-Cohort Sample CYD15 Case-Cohort Sample

*Average nAb titer = Geometric mean PRNT50 to the 4 dengue viruses in the vaccine construct

Cases = Dengue disease endpoint (VCD) between Month 13 and 25Controls = Never experienced VCD through Month 25

59 / 80

Page 93: The Search for an Optimal Immunological Surrogate Endpoint ...faculty.washington.edu/peterg/Sanofi2018/Slides/GilbertEstimated... · Surrogate Endpoint FrameworksOptimal Surrogate

Surrogate Endpoint Frameworks Optimal Surrogate Framework Simulation Studies Application to Dengue Trials Discussion

Month 13 PRNT50 Average Titers a Correlate of Risk inCYD14 (Moodie et al. 2017)Risk of DENV-Any Decreases with Month 13 Average

nAb Titer in CYD14

0

PRNT50Categories:

Low: ≤ 58 Med: 58−266 High: > 266

Vaccine Placebo

60 / 80

Page 94: The Search for an Optimal Immunological Surrogate Endpoint ...faculty.washington.edu/peterg/Sanofi2018/Slides/GilbertEstimated... · Surrogate Endpoint FrameworksOptimal Surrogate

Surrogate Endpoint Frameworks Optimal Surrogate Framework Simulation Studies Application to Dengue Trials Discussion

Month 13 PRNT50 Average Titers a Correlate of Risk inCYD15 (Moodie et al. 2017)

Risk of DENV-Any Decreases with Month 13 Average nAb Titer in CYD15

1

PRNT50Categories:

Low: ≤ 135 Med: 135−631 High: > 631

Vaccine Placebo

61 / 80

Page 95: The Search for an Optimal Immunological Surrogate Endpoint ...faculty.washington.edu/peterg/Sanofi2018/Slides/GilbertEstimated... · Surrogate Endpoint FrameworksOptimal Surrogate

Surrogate Endpoint Frameworks Optimal Surrogate Framework Simulation Studies Application to Dengue Trials Discussion

First Application: Inference on VE0 in CYD14

1 Obtain the TMLE-adjusted EOS from the CYD14 data(Wi ,Ai ,Si ,Yi ), i = 1, · · · , n

Surrogate endpoint ˆS0i = P0(Yi = 1 | |Wi ,Ai ,Si )

2 Based on this EOS, calculate point and confidence intervalestimates of

VE (ˆS0) = 1− Mean of ˆS0 Vaccine Group

Mean of ˆS0 Placebo Group

and of the numerator and denominator above

3 Compare these results to direct estimates of

VE0 = 1− Overall Disease Rate in the CYD14 Vaccine Group

Overall Disease Rate in the CYD14 Placebo Group

and of the numerator and denominator, based on the CYD14data (Wi ,Ai ,Yi ), i = 1, · · · , n

62 / 80

Page 96: The Search for an Optimal Immunological Surrogate Endpoint ...faculty.washington.edu/peterg/Sanofi2018/Slides/GilbertEstimated... · Surrogate Endpoint FrameworksOptimal Surrogate

Surrogate Endpoint Frameworks Optimal Surrogate Framework Simulation Studies Application to Dengue Trials Discussion

First Application: Inference on VE0 in CYD14

1 Obtain the TMLE-adjusted EOS from the CYD14 data(Wi ,Ai ,Si ,Yi ), i = 1, · · · , n

Surrogate endpoint ˆS0i = P0(Yi = 1 | |Wi ,Ai ,Si )

2 Based on this EOS, calculate point and confidence intervalestimates of

VE (ˆS0) = 1− Mean of ˆS0 Vaccine Group

Mean of ˆS0 Placebo Group

and of the numerator and denominator above

3 Compare these results to direct estimates of

VE0 = 1− Overall Disease Rate in the CYD14 Vaccine Group

Overall Disease Rate in the CYD14 Placebo Group

and of the numerator and denominator, based on the CYD14data (Wi ,Ai ,Yi ), i = 1, · · · , n

62 / 80

Page 97: The Search for an Optimal Immunological Surrogate Endpoint ...faculty.washington.edu/peterg/Sanofi2018/Slides/GilbertEstimated... · Surrogate Endpoint FrameworksOptimal Surrogate

Surrogate Endpoint Frameworks Optimal Surrogate Framework Simulation Studies Application to Dengue Trials Discussion

First Application: Inference on VE0 in CYD14

1 Obtain the TMLE-adjusted EOS from the CYD14 data(Wi ,Ai ,Si ,Yi ), i = 1, · · · , n

Surrogate endpoint ˆS0i = P0(Yi = 1 | |Wi ,Ai ,Si )

2 Based on this EOS, calculate point and confidence intervalestimates of

VE (ˆS0) = 1− Mean of ˆS0 Vaccine Group

Mean of ˆS0 Placebo Group

and of the numerator and denominator above

3 Compare these results to direct estimates of

VE0 = 1− Overall Disease Rate in the CYD14 Vaccine Group

Overall Disease Rate in the CYD14 Placebo Group

and of the numerator and denominator, based on the CYD14data (Wi ,Ai ,Yi ), i = 1, · · · , n

62 / 80

Page 98: The Search for an Optimal Immunological Surrogate Endpoint ...faculty.washington.edu/peterg/Sanofi2018/Slides/GilbertEstimated... · Surrogate Endpoint FrameworksOptimal Surrogate

Surrogate Endpoint Frameworks Optimal Surrogate Framework Simulation Studies Application to Dengue Trials Discussion

Second Application: Estimation of VE ∗ in CYD15 Basedon the Surrogate Built from CYD14

1 Calculate the ˆS∗i = P(Y ∗

i = 1 | |W ∗i ,A

∗i , S

∗i ) surrogate

endpoint values for CYD15 participants, i = 1, · · · , n∗

2 Use these to obtain point and confidence interval estimates ofthe CYD15 vaccine effect on the EOS and on the CYD15Vaccine Group and Placebo Group means of ˆS∗

Assume the three assumptions needed for valid bridging

3 Compare these results to direct estimates of

VE ∗ = 1− Overall Disease Rate in the CYD15 Vaccine Group

Overall Disease Rate in the CYD15 Placebo Group

and of the numerator and denominator, based on the CYD15data (W ∗

i ,A∗i ,Y

∗i ), i = 1, · · · , n∗

63 / 80

Page 99: The Search for an Optimal Immunological Surrogate Endpoint ...faculty.washington.edu/peterg/Sanofi2018/Slides/GilbertEstimated... · Surrogate Endpoint FrameworksOptimal Surrogate

Surrogate Endpoint Frameworks Optimal Surrogate Framework Simulation Studies Application to Dengue Trials Discussion

Second Application: Estimation of VE ∗ in CYD15 Basedon the Surrogate Built from CYD14

1 Calculate the ˆS∗i = P(Y ∗

i = 1 | |W ∗i ,A

∗i , S

∗i ) surrogate

endpoint values for CYD15 participants, i = 1, · · · , n∗

2 Use these to obtain point and confidence interval estimates ofthe CYD15 vaccine effect on the EOS and on the CYD15Vaccine Group and Placebo Group means of ˆS∗

Assume the three assumptions needed for valid bridging

3 Compare these results to direct estimates of

VE ∗ = 1− Overall Disease Rate in the CYD15 Vaccine Group

Overall Disease Rate in the CYD15 Placebo Group

and of the numerator and denominator, based on the CYD15data (W ∗

i ,A∗i ,Y

∗i ), i = 1, · · · , n∗

63 / 80

Page 100: The Search for an Optimal Immunological Surrogate Endpoint ...faculty.washington.edu/peterg/Sanofi2018/Slides/GilbertEstimated... · Surrogate Endpoint FrameworksOptimal Surrogate

Surrogate Endpoint Frameworks Optimal Surrogate Framework Simulation Studies Application to Dengue Trials Discussion

Second Application: Estimation of VE ∗ in CYD15 Basedon the Surrogate Built from CYD14

1 Calculate the ˆS∗i = P(Y ∗

i = 1 | |W ∗i ,A

∗i , S

∗i ) surrogate

endpoint values for CYD15 participants, i = 1, · · · , n∗

2 Use these to obtain point and confidence interval estimates ofthe CYD15 vaccine effect on the EOS and on the CYD15Vaccine Group and Placebo Group means of ˆS∗

Assume the three assumptions needed for valid bridging

3 Compare these results to direct estimates of

VE ∗ = 1− Overall Disease Rate in the CYD15 Vaccine Group

Overall Disease Rate in the CYD15 Placebo Group

and of the numerator and denominator, based on the CYD15data (W ∗

i ,A∗i ,Y

∗i ), i = 1, · · · , n∗

63 / 80

Page 101: The Search for an Optimal Immunological Surrogate Endpoint ...faculty.washington.edu/peterg/Sanofi2018/Slides/GilbertEstimated... · Surrogate Endpoint FrameworksOptimal Surrogate

Surrogate Endpoint Frameworks Optimal Surrogate Framework Simulation Studies Application to Dengue Trials Discussion

Super Learner Estimated Optimal Surrogate [CYD14]

Estimation of the Optimal Surrogate Based on a Randomized Trial 7

Figure 1. Point and 95% confidence interval estimates of cross-validated mean squared error (CV-MSE) for the vaccineand placebo groups of the CYD14 trial, for the top performing individual learners, the discrete super-learner, and the

super-learner.

Table 1Input variables, screens, and learner types used in the super-learner for the CYD14 dengue vaccine efficacy trial (35 total

statistical algorithms for estimating ψ0 = E0(Y |W, A, S) defined by screens crossed with learner types)

Input variables

W Baseline demographics age (range 2–14 years), sex,empirical frequencies of the 4 serotypes in placebo groupfailure events by country of the participant

S Month 13 seropositivity to each of the 4 serotypes in the CYD-TDVvaccine, and average, minimum, and maximum of the 4 titersfor both PRNT50 and Microneutralization Version 2 (V2) assays

Screens Boldfaced courier-font screens (e.g., screen.glmnet)available in the SuperLearner R package available at CRAN

screen.glmnet Include variables with non-zero coefficients in astandard implementation of SL.glmnet (i.e., lasso)

screen.univar.logistic.x Univariate logistic regression p-value < 0.10using “x” most univariatly significant terms.

screen.corX.x Disallow pairs of quantitative variables with R2 > “0.x′′

screen.PRNT Disallow Microneutralization V2 titer variablesscreen.MNv2 Disallow PRNT50 titer variables

Learner types Boldfaced courier-font learning algorithms (e.g., SL.mean) areavailable in the SuperLearner R package available at CRAN

SL.mean E0(Y |W, A = a, S)a = βa for a ∈ {0, 1}SL.glm Logistic regression with all input variablesSL.step Best logistic regression model by AIC from a step-wise searchSL.bayesglm Logistic regression utilizing Cauchy Bayesian priors on model

parametersSL.polymars Multivariate adaptive polynomial spline regressionDiscrete SL van der Laan, Polley, and Hubbard (2007)Super Learner (SL) van der Laan, Polley, and Hubbard (2007)

Note: a All learners were fit separately for each treatment group A = a for a ∈ {0, 1} as described in Section 6.1. This is explicitly statedhere for SL.mean.

64 / 80

Page 102: The Search for an Optimal Immunological Surrogate Endpoint ...faculty.washington.edu/peterg/Sanofi2018/Slides/GilbertEstimated... · Surrogate Endpoint FrameworksOptimal Surrogate

Surrogate Endpoint Frameworks Optimal Surrogate Framework Simulation Studies Application to Dengue Trials Discussion

Comparison of Prediction Performance Across Estimators(CV-AUCs∗): CYD14

Cross-Validated Area under the ROC Curves (CV-AUCs*) for Statistical Algorithms: Demo + MNv2 + PRNT

1

*95% CIs for CV-AUC from method of van der Laan, Hubbard, and Pajouh (2013)∗Cross-validated area under the ROC curve (Van der Laan, Hubbard, and Pajouh, 2013) 65 / 80

Page 103: The Search for an Optimal Immunological Surrogate Endpoint ...faculty.washington.edu/peterg/Sanofi2018/Slides/GilbertEstimated... · Surrogate Endpoint FrameworksOptimal Surrogate

Surrogate Endpoint Frameworks Optimal Surrogate Framework Simulation Studies Application to Dengue Trials Discussion

Adding Month 13 Neutralization Markers ImprovesPrediction Over Baseline Demographics Only: CYD14

Most Accurate Model Includes Both PRNT and MNv2 Assays; MNv2-Only Almost as Good

3

Cross-Validated ROC Curves*

D1: DemographicsD2: Demo + MNv2D3: Demo + PRNTD4: Demo + MNv2 + PRNT

*van der Laan, Hubbard, and Pajouh (2013)

Models based on Demo only

Models includingneutralization markers

(FP = No VCD but predicts VCD)

66 / 80

Page 104: The Search for an Optimal Immunological Surrogate Endpoint ...faculty.washington.edu/peterg/Sanofi2018/Slides/GilbertEstimated... · Surrogate Endpoint FrameworksOptimal Surrogate

Surrogate Endpoint Frameworks Optimal Surrogate Framework Simulation Studies Application to Dengue Trials Discussion

Using Both Assays Improves Performance in the RelevantRange of False Positive Rates: CYD14Using Both Assays Improves Sensitivity to Detect VCD

in the Relevant Range of False Positive Rates

Both MNv2and PRNT

4

(FP = No VCD but predicts VCD)

Zooming in on the lower-left of the figure

67 / 80

Page 105: The Search for an Optimal Immunological Surrogate Endpoint ...faculty.washington.edu/peterg/Sanofi2018/Slides/GilbertEstimated... · Surrogate Endpoint FrameworksOptimal Surrogate

Surrogate Endpoint Frameworks Optimal Surrogate Framework Simulation Studies Application to Dengue Trials Discussion

Predicted Risks of DENV-Any for the Vaccine Group onHeld-Out Data: CYD14Predicted DENV-Any Risks for the Vaccine

Group Best Models on Held-Out Data

5

68 / 80

Page 106: The Search for an Optimal Immunological Surrogate Endpoint ...faculty.washington.edu/peterg/Sanofi2018/Slides/GilbertEstimated... · Surrogate Endpoint FrameworksOptimal Surrogate

Surrogate Endpoint Frameworks Optimal Surrogate Framework Simulation Studies Application to Dengue Trials Discussion

Best Performing Models for Estimating P0(Y | W ,A, S):CYD14

Estimation of the Optimal Surrogate Based on a Randomized Trial 9

Table 2Best performing models for estimating ψ0 = E0(Y |W, A, S) for the vaccine and placebo groups of the CYD14 trial. For boththe vaccine and placebo groups the model with the lowest CV-MSE was a logistic regression (glm) using variables selected

from the screen screen.MNv2 in Table 1.

Model term Coefficient Odds ratio 2-Sided P-value

Vaccine model

(Intercept) 1.09 2.96 0.26AGE.9.11 −0.09 0.91 0.74AGE.12.14 −2.46 0.09 <0.01MALE −0.36 0.70 0.09M13.MNv2.S1b −3.62 0.03 <0.01M13.MNv2.S2 0.77 2.16 0.02M13.MNv2.S3 1.41 4.09 0.04M13.MNv2.S4 −0.12 0.89 0.81M13.MNv2.Avec 3.45 31.53 <0.01M13.MNv2.Min −3.53 0.03 <0.01M13.MNv2.Max −0.59 0.55 0.28Sero2.frequencyd −0.91 <0.01 <0.01Sero3.frequency −0.57 <0.01 <0.01Sero4.frequency −0.38 0.02 <0.01

Placebo model

(Intercept) 1.97 7.16 0.01AGE.9.11 0.84 2.32 <0.01AGE.12.14 −0.17 0.85 0.55MALE 0.04 1.04 0.82M13.MNv2.S1b −1.10 0.33 <0.01M13.MNv2.S2 0.25 1.29 0.34M13.MNv2.S3 0.56 1.76 0.10M13.MNv2.S4 0.06 1.06 0.84M13.MNv2.Avec 1.01 2.75 0.43M13.MNv2.Min −2.62 0.07 <0.01M13.MNv2.Max −0.25 0.78 0.51Sero2.frequencyd −0.72 <0.01 <0.01Sero3.frequency −0.54 <0.01 <0.01Sero4.frequency −0.46 <0.01 <0.01

Notes: a The reference age category is 2–8 year olds; b M13.MNv2.S1 is the binary indicator of a Month 13 positive response to serotype

1 using the MNv2 assay, with positive response defined by MNv2 serotype neutralization titer ≥ 10. M13.MNv2.S2-M13.MNv2.S4

are defined similarly; c M13.MNv2.Ave, M13.MNv2.Min, and M13.MNv2.Max coefficients are per one log10 increase in neutraliza-

tion titer value; d Serotype frequency variable coefficients are per 0.10 increase in the estimated serotype frequency of a participant’s

country.

avoids the surrogate paradox, which can have disastrous con-sequences. Yet, VanderWeele (2013) shows that commonlyused methods for surrogate endpoint evaluation generally donot guarantee avoiding this paradox. The first useful featureof the newly proposed approach is that it starts at this min-imal requirement, defining the optimal surrogate in a wayguaranteed to satisfy the Prentice definition of a valid surro-gate within the original trial and thus avoid the paradox (andthen the estimated optimal surrogate (EOS), which can beused as a surrogate endpoint in practice, satisfies the Prenticedefinition in large samples). As such the proposed approachresponds to Pearl’s (2013) question: “If we take the negationof the “surrogate paradox” as a criterion for “good” surrogate,why cannot we create a new, formal definition of “surrogacy”that (1) will automatically avoid the paradox?...” A seconduseful feature of the approach is that the treatment effect on

the EOS has the same interpretation as the treatment effecton the clinical endpoint of interest.

A third useful feature of the proposed approach is that theEOS– in being built by super-learner followed by a TMLEupdate– contains all information about the average clinicaltreatment effect in the original trial. A fourth useful fea-ture is the approach’s use of super-learner with its principledcross-validation approach to build and compare best modelsfor estimating the optimal surrogate. Super-learner is use-ful for applications where multiple baseline covariates and/orintermediate response endpoints are measured, yet there isconsiderable uncertainty about how to best predict the studyoutcome from these collected data. Moreover, while we havefocused on randomized studies, this framework also appliesfor generating promising candidate surrogates based on obser-vational studies, with all of the results holding under the

69 / 80

pgilbert
Highlight
pgilbert
Highlight
pgilbert
Highlight
Page 107: The Search for an Optimal Immunological Surrogate Endpoint ...faculty.washington.edu/peterg/Sanofi2018/Slides/GilbertEstimated... · Surrogate Endpoint FrameworksOptimal Surrogate

Surrogate Endpoint Frameworks Optimal Surrogate Framework Simulation Studies Application to Dengue Trials Discussion

Estimated Optimal Surrogate (EOS) TMLEs of TargetParameters: CYD14

Parameter TMLE Based on EOS TMLE Based on (W ,A,Y )P0(Y = 1|Vac) 0.017 (0.016–0.019) 0.017 (0.014–0.021)P0(Y = 1|Plc) 0.039 (0.036–0.042) 0.039 (0.031–0.047)

VE0 = 1− P0(Y=1|Vac)P0(Y=1|Plc) 55% (49–61) 55% (40–66)

The point estimate results have to be similar by construction!

71 / 80

Page 108: The Search for an Optimal Immunological Surrogate Endpoint ...faculty.washington.edu/peterg/Sanofi2018/Slides/GilbertEstimated... · Surrogate Endpoint FrameworksOptimal Surrogate

Surrogate Endpoint Frameworks Optimal Surrogate Framework Simulation Studies Application to Dengue Trials Discussion

Using the Estimated Optimal Surrogate (EOS) in CYD15

How well do the EOS values ˆS∗i predict Y ∗

i in CYD15?10 Biometrics

Figure 2. (a) Empirical reverse cumulative distribution functions (cdfs) of the estimated optimal surrogateψ#

n (Wi, Ai = a, Si) for the CYD14 trial by vaccine/placebo assignment A = a ∈ {0, 1} and dengue outcome case/control statusY = y ∈ {0, 1}. (b) Empirical reverse cdfs of ψ#

n (W∗i , A∗

i = a, S∗i ) for CYD15 participants by vaccine/placebo assignment

A∗ = a ∈ {0, 1} and dengue outcome case/control status Y ∗ = y ∈ {0, 1}, where ψ#n (·) was estimated from the CYD14 trial

data. The results show that the surrogate better classifies dengue outcomes of participants in the original trial than in thenew trial (not surprisingly).

additional (challenging) assumption that all confounders W

of treatment assignment are measured and included in thesuper-learner.

A challenge posed to the framework is that through super-learner the EOS may be based on a complicated combinationof models that is hard to interpret. This underscores theimportance of building multiple EOSs from different inputvariable sets ranging from single-variable to all-variable mod-els, where cross-validation criteria allow principled selection ofa most parsimonious EOS with near-optimal predictive per-formance. A related challenge is that researchers in futuretrials may not have access to the code used by the pre-vious researchers to calculate the EOS. This may require

use of an open research paradigm where web calculators aremade available that input (W, A, S) values and output EOSvalues.

This article considers an ideal setting with no missing dataand where the clinical outcome is never observed before theintermediate response endpoints are measured. Moreover, weused a particular loss function for defining optimal predic-tion. Future work is of interest to accommodate these issues.Theorems 2 and 3 provide conditions for using the EOS froman original trial to confer correct estimation of the clinicaltreatment effect in a new setting/trial based on this surro-gate endpoint without measuring the clinical endpoint. Theinference part of these results hold for an infinite original trial,

Table 3Comparison of inferences on the surrogate parameters θa

ψ#n

(P) ≡ EP

(EP(ψ#

n (W∗, a, S∗) | W∗, A∗ = a))

for each a ∈ {0, 1} and

VEψ#n

(P) = 1 − θ1

ψ#n

(P)/θ0

ψ#n

(P) based on (W∗, A∗, ψ#n (W∗, A∗, S∗)) versus direct inferences on the clinical dengue endpoint

parameters EP(Y ∗a ) and θ∗

P = VE∗P = 1 − EP(Y ∗

1)/EP(Y ∗0) in CYD15. Included is a summary of enrollment numbers, incidence

of VCD, and number of participants with measured titers for each study.

Surrogate parameters Clinical parametersestimated by TMLEsa estimated by TMLEsb

θ1

ψ#n

(P) 0.020 (95% CI 0.017–0.022) EP(Y ∗1) 0.014 (95% CI 0.012–0.017)

θ0

ψ#n

(P) 0.057 (95% CI 0.049–0.065) EP(Y ∗0) 0.037 (95% CI 0.031–0.043)

VEψ#n

(P) 66% (95% CI 58–72) VE∗P 61% (95% CI 51–69)

No. Enrolled No. VCD cases No. with (W, A, S) or(Y = 1 or Y ∗ = 1) (W∗, A∗, S∗) measuredc

Study Vaccine, Placebo Vaccine, Placebo Vaccine, Placebo

CYD14 6851, 3424 117, 133 736, 415CYD15 13920, 6949 184, 232 944, 587

Notes: a TMLEs θTMLE,1

ψ#n

(P), θTMLE,0

ψ#n

(P), and VETMLE

ψ#n

(P) = 1 − θTMLE,1

ψ#n

(P)/θTMLE,0

ψ#n

(P); b TMLEs θTMLE,1n∗ (P), θ

TMLE,0n∗ (P), and VEn∗(P) =

1 − θTMLE,1n∗ (P)/θTMLE,0

n∗ (P); c Measured in 98.3% and 99.8% of endpoint cases with Y = 1 or Y∗ = 1 for CYD14 and CYD15, respectively.

72 / 80

Page 109: The Search for an Optimal Immunological Surrogate Endpoint ...faculty.washington.edu/peterg/Sanofi2018/Slides/GilbertEstimated... · Surrogate Endpoint FrameworksOptimal Surrogate

Surrogate Endpoint Frameworks Optimal Surrogate Framework Simulation Studies Application to Dengue Trials Discussion

How Well Does the Surrogate-Based Estimator EstimateVE ∗ in CYD15?

Table: Estimation in CYD15 based on the EOS built in CYD14 (notusing outcome data Y ∗ in CYD15) vs. TMLE estimation using(W ∗,A∗,Y ∗) in CYD15

TMLEs of TMLEs ofSurrogate Parameters1 Clinical Parameters2

Mean ˆS∗ Vac 0.020 (0.017–0.022) P(Y ∗ = 1|Vac) 0.014 (0.012–0.017)

Mean ˆS∗ Plc 0.057 (0.049–0.065) P(Y ∗ = 1|Plc) 0.037 (0.031–0.043)

VE on ˆS∗ 66% (58–72) VE∗ 61% (51–69)

1Based on (W ∗i ,A

∗i ,

ˆS∗i (W ∗

i ,A∗i ,S

∗i ))

2Based on (W ∗i ,A

∗i ,Y

∗i ) [use the actual clinical data]

73 / 80

Page 110: The Search for an Optimal Immunological Surrogate Endpoint ...faculty.washington.edu/peterg/Sanofi2018/Slides/GilbertEstimated... · Surrogate Endpoint FrameworksOptimal Surrogate

Surrogate Endpoint Frameworks Optimal Surrogate Framework Simulation Studies Application to Dengue Trials Discussion

1 Surrogate Endpoint Frameworks

2 Optimal Surrogate Framework

3 Simulation Studies

4 Application to Dengue Trials

5 Discussion

74 / 80

Page 111: The Search for an Optimal Immunological Surrogate Endpoint ...faculty.washington.edu/peterg/Sanofi2018/Slides/GilbertEstimated... · Surrogate Endpoint FrameworksOptimal Surrogate

Surrogate Endpoint Frameworks Optimal Surrogate Framework Simulation Studies Application to Dengue Trials Discussion

Start at the Right Place

VanderWeele (2013, Biometrics) and discussants Joffe (2013)and Pearl (2013) suggest that a minimal requirement for anintermediate endpoint to be a useful surrogate endpoint isthat it avoids the surrogate paradox

VanderWeele (2013) shows that commonly used methods forsurrogate endpoint evaluation generally do not guaranteeavoiding this paradox

The optimal surrogate approach starts at this minimalrequirement, defining the optimal surrogate in a wayguaranteed to satisfy the Prentice definition of a validsurrogate

Responds to Pearl’s (2013) question:“If we take the negation of the “surrogate paradox” as acriterion for “good” surrogate, why cannot we create anew, formal definition of “surrogacy” that willautomatically avoid the paradox?...”

75 / 80

Page 112: The Search for an Optimal Immunological Surrogate Endpoint ...faculty.washington.edu/peterg/Sanofi2018/Slides/GilbertEstimated... · Surrogate Endpoint FrameworksOptimal Surrogate

Surrogate Endpoint Frameworks Optimal Surrogate Framework Simulation Studies Application to Dengue Trials Discussion

Start at the Right Place

VanderWeele (2013, Biometrics) and discussants Joffe (2013)and Pearl (2013) suggest that a minimal requirement for anintermediate endpoint to be a useful surrogate endpoint isthat it avoids the surrogate paradox

VanderWeele (2013) shows that commonly used methods forsurrogate endpoint evaluation generally do not guaranteeavoiding this paradox

The optimal surrogate approach starts at this minimalrequirement, defining the optimal surrogate in a wayguaranteed to satisfy the Prentice definition of a validsurrogate

Responds to Pearl’s (2013) question:“If we take the negation of the “surrogate paradox” as acriterion for “good” surrogate, why cannot we create anew, formal definition of “surrogacy” that willautomatically avoid the paradox?...”

75 / 80

Page 113: The Search for an Optimal Immunological Surrogate Endpoint ...faculty.washington.edu/peterg/Sanofi2018/Slides/GilbertEstimated... · Surrogate Endpoint FrameworksOptimal Surrogate

Surrogate Endpoint Frameworks Optimal Surrogate Framework Simulation Studies Application to Dengue Trials Discussion

Start at the Right Place: Interpretability

The proposed approach uses predicted clinical endpoint valuesas the surrogate, implying that the mean surrogate treatmenteffect has the same interpretation as the mean clinicaltreatment effect

An obvious approach for maximally tying the surrogate to theclinical endpoint

Yet typically surrogate endpoints are biomarkers on their ownscale, which is often different from the clinical endpoint scale

76 / 80

Page 114: The Search for an Optimal Immunological Surrogate Endpoint ...faculty.washington.edu/peterg/Sanofi2018/Slides/GilbertEstimated... · Surrogate Endpoint FrameworksOptimal Surrogate

Surrogate Endpoint Frameworks Optimal Surrogate Framework Simulation Studies Application to Dengue Trials Discussion

Nonparametric Supervised Learning Approach

Using super-learner + TMLE seeks to avoid dubiousassumptions and use all of the information in the data

Broad application to clinical fields where multiple biomarkersare measured that could contribute to a surrogate endpoint,and the objective is supervised learning of most promisingsurrogate endpoints that may depend on baseline covariatesas well as post-vaccination response response endpoints

This framework also applies for exploratory analyses ofobservational studies to generate promising candidatesurrogates, with all of the results holding under the additionalassumption that all confounders W of treatment assignmentare measured and included in the super-learner

77 / 80

Page 115: The Search for an Optimal Immunological Surrogate Endpoint ...faculty.washington.edu/peterg/Sanofi2018/Slides/GilbertEstimated... · Surrogate Endpoint FrameworksOptimal Surrogate

Surrogate Endpoint Frameworks Optimal Surrogate Framework Simulation Studies Application to Dengue Trials Discussion

Nonparametric Supervised Learning Approach

Using super-learner + TMLE seeks to avoid dubiousassumptions and use all of the information in the data

Broad application to clinical fields where multiple biomarkersare measured that could contribute to a surrogate endpoint,and the objective is supervised learning of most promisingsurrogate endpoints that may depend on baseline covariatesas well as post-vaccination response response endpoints

This framework also applies for exploratory analyses ofobservational studies to generate promising candidatesurrogates, with all of the results holding under the additionalassumption that all confounders W of treatment assignmentare measured and included in the super-learner

77 / 80

Page 116: The Search for an Optimal Immunological Surrogate Endpoint ...faculty.washington.edu/peterg/Sanofi2018/Slides/GilbertEstimated... · Surrogate Endpoint FrameworksOptimal Surrogate

Surrogate Endpoint Frameworks Optimal Surrogate Framework Simulation Studies Application to Dengue Trials Discussion

Some Challenges Posed to the Framework

The estimated optimal surrogate (EOS) may be based on acomplicated combination of models that is hard to interpret

Hence the importance of building multiple EOSs from differentinput variable sets ranging from single-variable to all-variablemodels, where cross-validation criteria allow principledselection of a most parsimonious EOS with acceptableperformance

Will it be complicated for other researchers to use an EOS?

Broad use may require a research paradigm embracing openresearch that posts to the web a calculator that inputs(W ,A,S) and outputs the EOS value

78 / 80

Page 117: The Search for an Optimal Immunological Surrogate Endpoint ...faculty.washington.edu/peterg/Sanofi2018/Slides/GilbertEstimated... · Surrogate Endpoint FrameworksOptimal Surrogate

Surrogate Endpoint Frameworks Optimal Surrogate Framework Simulation Studies Application to Dengue Trials Discussion

Some Challenges Posed to the Framework

The estimated optimal surrogate (EOS) may be based on acomplicated combination of models that is hard to interpret

Hence the importance of building multiple EOSs from differentinput variable sets ranging from single-variable to all-variablemodels, where cross-validation criteria allow principledselection of a most parsimonious EOS with acceptableperformance

Will it be complicated for other researchers to use an EOS?

Broad use may require a research paradigm embracing openresearch that posts to the web a calculator that inputs(W ,A,S) and outputs the EOS value

78 / 80

Page 118: The Search for an Optimal Immunological Surrogate Endpoint ...faculty.washington.edu/peterg/Sanofi2018/Slides/GilbertEstimated... · Surrogate Endpoint FrameworksOptimal Surrogate

Surrogate Endpoint Frameworks Optimal Surrogate Framework Simulation Studies Application to Dengue Trials Discussion

Elaborations

Missing data on (W ,A,S ,Y )

E.g., case-cohort or case-control sampling of SHappenstance missing data

Some participants experience Y before S is measured at τ

Right-censoring of Y (failure time endpoint), competing risksoutcomes

Tailoring the super-learner to contextual features [sample size,event rate, dimensionality of (W , S)]

Confidence intervals about the clinical treatment effectVE ∗ = 1− P(Y ∗ = 1|A = 1)/P(Y ∗ = 1|A = 0) in a newsetting accounting for the error in estimating the optimalsurrogate

79 / 80

Page 119: The Search for an Optimal Immunological Surrogate Endpoint ...faculty.washington.edu/peterg/Sanofi2018/Slides/GilbertEstimated... · Surrogate Endpoint FrameworksOptimal Surrogate

Surrogate Endpoint Frameworks Optimal Surrogate Framework Simulation Studies Application to Dengue Trials Discussion

Acknowledgements

SanofiPasteur colleagues for collaboration and sharing thedata

Participants and study personnel of the CYD14 and CYD15dengue Phase 3 trials

NIH NIAID support for the grant “Statistical Methods in HIVVaccine Efficacy Trials”

80 / 80