Randomization: Too Important to Gamble with

Optimizing Clinical Trials: Concept to Conclusion™ © 2012 Medidata Solutions, Inc. 1

Optimizing Clinical Trials:Concept to Conclusion™

Randomization: Too Important to Gamble with

A Presentation for the Delaware Chapter of the ASAOct 18, 2012

Dennis Sweitzer, Ph.D., Principal Biostatistician

Medidata Randomization Center of Excellence


Outline

Randomized Controlled Trials• Basics• BalanceRandomization methods • Complete Randomization• Strict Minimization• Permuted Block• Dynamic Allocation (Covariate-adaptive, not Response-Adaptive)Randomization Metrics• Balance• Predictability• Loss of Power /Loss of Efficiency• Secondary Imbalance: drop-outsSimulations comparing methods• Confounding site & treatment effects (small sites)• Overall performance• Discontinuing patients• Weighting stratification factorsMeta-Balance


Why randomize anyway?Some basic principles


Why Gold Standard?

Randomized Controlled Trial

• Trial: Prospective & Specific• Controlled:

• Comparison with Control group • (placebo or active)

• Controlled procedures ⇒ Only Test Treatment Varies

• Randomization: Minimizes biases• Allocation bias • Selection bias • Permits blinding


Eliminating Bias

¿ The Fact of bias ?• (conscious, unconscious, or instinctive)

¿ The Question of bias ?• Always 2nd guessing• Critics will think of unanticipated things

¡ Solution !• Treat it as a game• 1 statistician vs N clinicians• Statistician generates a random sequence• Clinicians sequential guess at each assignment• Statistician wins if clinician guesses are no better than chance

(NB: 75% wrong is just as bad as 75% right)


Randomization Metrics

What do we want in a randomization sequence or system?Randomness Unpredictable

⟶ Reduce Allocation Bias (All studies)

⟶ Reduce Selection Bias (All studies)

⟶ Reduce placebo effects (Blinded studies)

Balance “Loss of Efficiency”⟶ Maximizes statistical power

⟶ Minimize Confounding

⟶ Enhance Credibility (Face Validity)


Balancing


Balanced Study

Control Test

Equal allocation between treatment arms• Maximizes

Statistical Power


ImbalancedStatistical power limited by

smallest arm

• 36 subject simulation with Complete Randomization

⟶average loss ≈ 1 subject

10% lose ≥2 subject• Can add 2 to compensate

• BUT only large imbalances have much effect

on statistical power

Resulting in light weight results….

Severe Imbalances are rare in large studies Pr{worse than 60:40 split} for:

• n=25 <42% n=100 <4.4% n=400 0.006%⟶ ⟶ ⟶


(NB: Planned Imbalance)

1:1 randomization maximizes power per patientBut there are other considerations

• Utility:• Need 100 patients on drug to monitor safety• Study only requires 60 (30/arm)• 2:1 randomization ⟶ 100 Test & 50 Placebo

• Motivation:• Better enrollment if 75% chance of Test drug (3:1)

• Ethics:• 85 Placebo + 255 Test vs. 125 Placebo + 125 Test


Imbalance

• Overall balance• Only an issue for small studies

• Subgroup Balance• Fixed size studies can have variable sized subgroups

⟶ Increased risk of underpowered subgroups


Effective Loss of Sample Size

Effective Loss = Reduction of Power as Reduction in Sample Size

Simulations of: • 36 and 18 subjects,

• males as strata at 33% of population, • randomized 1:1

• (complete randomization)

Pla

Test

Females

Con

Test

Males


Bad Imbalance!

Treatment Imbalances within factors ⟶ spurious findings…..

Pla

Test

Females

Pla

Test

Males

Leads to conversations like:

Higher estrogen levels in patients

on Test Treatment ??

ANCOVA showed no

differences in estrogen

levels due to treatmentHmm…

?

Credibility…..


Randomization Methods

(See Animated Powerpoint Slides…)


Randomization

4 methods• Complete Randomization (classic approach)• Strict Minimization • Permuted Block (frequently used)• Dynamic Allocation (gaining in popularity)


Complete Randomization

Every assignment• Same probability for each assignment• Ignore Treatment Imbalances• No restrictions on treatment assignmentsAdvantages:• Simple• Robust against selection & accidental bias• Maximum UnpredictabilityDisadvantage• High likelihood of imbalances (smaller samples)

.


Minimization

Strict Minimization randomizes to the imbalanced arm


Minimization

Strict Minimization rebalances the Arms• BUT at a cost in

predictability• Random only when treatments are currently

balanced


Permuted Block

Blocks of Patients (1, 2, or 3 per treatment)

Here: 2:2 Allocation

• Balanced

• Some Predictability

T P P ?

T P P T

T P P *

(Unless Incomplete Blocks:

More strata

More incomplete⟶ )


Dynamic Allocation

Biases Randomization to the imbalanced arm

• Unpredictable• Almost Balanced


Dynamic Allocation

Complete Randomization• Optimizes Unpredictability

• Ignores Balance

Strict Minimization• Optimizes Balance

• Ignores Predictability

Dynamic Allocation 2nd Best Probability Parameter

Controls Balance vs. Predictability

Tradeoff


Dynamic Allocation Flexibility

2nd Best Probability= 0

⟶ Strict Minimization


Dynamic Allocation Flexibility

2nd Best Probability= 0.5

⟶ Complete Randomization

(for 2 treatment arms)


Stratification Factors


Stratification Factors

Over all Ages:

Pla

Test

Pla

Test

Pla Test

Over both sexes

PlaTest

PlaTest

PlaTest

PlaTest

Pla Test

Males Females

18-35 yo

35-65 yo

>65 yo

Pla Test

PlaTest

PlaTest

Factors Main Effects≣Strata 1≣ st Order Interactions

Randomizing a 25 yo Male:To PLA

Worsens Male ⟶balance

To Test Worsens 18-⟶

35yo balance

Balance

w/in

6

Strata

?

Marginal

Balance

Marginal

Balance

Pla Test

Overall Balance


Permuted Block Stratified Randomization

• Only balances within strata

• Most strata will have incomplete blocks

• Imbalances accumulate at margins

Males Females

18-35 yo

35-65 yo

>65 yo

T P P T

P T * *

P * * *

P * * *

T T P *

T P T *

Pla Test

Over both sexes

Pla

Test

PlaTest

Over all Ages:

Pla

Test

Pla

Test

Pla Test


Minimization & Marginal Balance

* Only balances on margins

* Useful if too many strata, e.g.:

* Appropriate for a main effects analysis (ie, no interactions)

* *

Balance

w/in

6 S

trata

?

Marginal Balance

Marginal Balance

Over all Ages:

Pla

Test

Pla

Test

Pla Test

Over both sexes

PlaTest

PlaTest

PlaTest

PlaTest

Pla Test

Males Females

18-35 yo

35-65 yo

>65 yo

Pla Test

PlaTest

PlaTest

Pla Test

Overall Balance


Stratification & Dynamic Allocation

Over all Ages:

Pla

Test

Pla

Test

Pla Test

Over both sexes

Pla

Test

PlaTest

Pla

Test

PlaTest

Pla Test

Males Females

18-35 yo

35-65 yo

>65 yo

Pla Test

PlaTest

PlaTest

DA: uses weighted combination

of

• Overall balance

• Marginal balances

• Strata balance

⇒ Flexible

Balance

w/in

6 S

trata

?

Marginal Balance

Marginal Balance

Pla Test

Overall Balance


Site as a Special Subgroup(Max 2 lines, 35 characters)


Imbalance

• Overall balance• Only an issue for small studies

• Subgroup Balance• Fixed size studies can have variable sized subgroups

⟶ Increased risk of underpowered subgroups

• Site as special case of subgroup• Small sites ⟶ Increased risk of "monotherapy” at site

⟶ Confounding site & treatment effects ⟶ Effectively non-informative/”lost” patients

• Actual vs Assumed distribution of site size


Enrollment per Center (Densities)

Data Sample• 13 Studies• 7.7 mo Average Enrollment period • 3953 Obs.Pts • 460 Listed Sites • 372 Active.Sites

Size Categories: {0, 1, 2, 3, 4-7, 8-11, 12-15, 16-19, 20-29, 30-39, 40-49, 50-59, 60-79, 80-99, 100-149, 150-199, ≥200 }


Enrollment per Site (#Sites)

Data Sample• 13 Studies• 7.7 mo Average Enrollment period • 3953 Obs.Pts • 460 Listed Sites • 372 Active.Sites# Sites per Size Category {0, 1, 2, 3, 4-7, 8-11, 12-15,

16-19, 20-29, 30-39, 40-49, 50-59, 60-79, 80-99, 100-149, 150-199, ≥200 }


Site Enrollment Simulation

Simulation based on Observations• 4 mo Enrollment Period• Enrollment ~ Poisson distribution

μ = Obs. Pts/mo (active sites) or

μ ≈ 0.5 / Enrollment period (non-active sites)

• Randomize using CR, PB(2:2), or DA(0.15).• Confounded Pts ≣ Patients at centers with only one

treatment ⇒ treatment & center effects are confounded


Results

• Affected studies had many sites with low enrollment• Studies with fewer sites (and more pts at each) were rarely affected• Dynamic Allocation reduced confounding slightly more effectively than

permuted block

mean ±SD (80%

C.I.)



How do we measure “badness” of a randomization sequence or system?• Predictability

• Goal: an observer can guess no better than chance

⟶ Score based on Blackwell-Hodges guessing rule• Easily calculated

• ImbalanceImbalance ⟶ reduced statistical power

⟶ “Loss of Efficiency”• Measure as effective loss in number of subjects


Blackwell-Hodges

Use Blackwell-Hodges guessing rule• Directly corresponds to game interpretation• Investigator always guesses the most probable treatment

assignment, based on past assignments• “ bias factor F”

F ≣ abs(# Correct – Expected # Correct by chance alone)• Measures potential for selection bias• Modifications:

• Limits on knowledge of investigator (eg, can only know prior treatment allocation on own site)

• Score as percentage

e.g., Score ≣ abs(% Correct – 50%)


Blackwell-Hodges Scoring (1)

For treatment sequence “TCCC”

Initial guess ⟶ Expectation = ½ “T” ⟶ Imbalance =+1 ⟶ Guess C ⟶ Correct “TC” ⟶ Imbalance=0 ⟶ Guess either

⟶ Expectation=½ “TCC” ⟶ Imbalance=-1 ⟶ Guess T ⟶ Wrong“TCCC” ⟶ # Correct= ½ + 1+ ½ +0 =2

Score = #Correct - 2 = 2-2 = 0



For treatment sequence “TCCC”“TCCC” # Correct= ½ + 1+ ½ +0 =2⟶

Complete Randomization Pr{“TCCC”} = 1/16⇒

Dynamic Allocation (p=0.15) Pr{“TCCC”}= 0.5 *0.85 * 0.5 * 0.15 = 0.031875⇒

Permuted Block (length≤4) PR{“TCCC”} = 0⇒Strict Minimization Pr{“TCCC”}=0⇒



Sequence “TCCT”# Correct= ½ + 1 + ½ + 1 = 3

Score = 3 – 2 = 1

• Complete Randomization Pr{TCCT}= 1/16⇒• Strict Minimization Pr{TCCT} = ½*1*½*1 = ¼ ⇒• Permuted Block Pr{TCCT} = 1/6⇒

(NB: 6 permutations of TTCC)• Dynamic Allocation (2nd best prob.=0.15)

Pr{TCCT} = 0.5 * 0.85* 0.5 * 0.85 = 0.180625⇒


Warning!

Blackwell-Hodges • Assesses potential selection bias

― Given known imbalance!¿¿ But which imbalance(s)??(Overall imbalance? Within strata? Within Factors?)

• Henceforth: only use imbalance within strata• Proxy for center • Assume observer only knows

imbalance within “his center”• Simple & unambiguous

M Requires some caution in interpretation

Local Predictability

ONLY


Loss of Efficiency

• Loss can be expressed as equivalent # Patients

• In a 100 patient study:Loss of Efficiency= 5

⇒ A perfectly designed study would require only 95

Inference in Covariate-Adaptive allocation

Elsa Valdes Marquez & Nick Fieller

EFSPI Adaptive Randomisation Meeting

Brussels, 7 December 2006

http://www.efspi.org/PDF/activities/international/adaptive-rando-docs/2ValdesMarquez.pdf





RCT vs DOE

Designed Experiment (DOE): ⟶ Select z and covariate values to minimize Ln

RCT Select only ⟶ z (No control of covariates)

X ≣ design matrix:

⟶n rows, 1 per pt ⟶K columns, 1 per covariate

z ≣ Treatment assignments


Loss of Efficiency (Máquez & Fieller)

Dynamic Allocation

Sequentially assign Z to minimize


Loss of Efficiency (Máquez & Fieller)


Randomization Performance Simulations


Simulation Set up

3 methods:• Complete Randomization• Permuted Block• Dynamic Allocation

Each simulated patient randomized w/ each method

4 Measures:• Loss of Efficiency • B-H Score (“Within Strata”)• Overall Imbalance• Relative Loss of Efficiency vs CR• % Loss of Efficiency (of #pts)

6 Strata (Factors: Sex, Age)• 33% or 50% Males• 1:1:1, 1:1:2, 1:2:3

(Young : Middle : Old)

• 48 subjects Total• With random 25% Dropout


Note on Figures

Plot B-H score

vs

Loss of Efficiency

Median

+

80% C.I. ⇒

10% lower& 10% higher


Simulation Results(1)

⟵Averages of Metrics

But for managing risk, need Worst Case

80% ⟶ Confidence Intervals

Predictability %Imbalance Efficiency LossDA(0.00) 22% 0.6% 0.87DA(0.15) 16% 1.6% 1.45DA(0.25) 13% 2.8% 1.99DA(0.33) 8% 4.3% 2.64DA(0.50) 4% 11.3% 4.99CR 4% 11.4% 5.03PB(8:8) 7% 7.1% 3.00PB(4:4) 13% 4.9% 1.52PB(3:3) 16% 4.2% 1.13PB(2:2) 19% 3.5% 0.79PB(1:1) 23% 2.6% 0.47

Both DA & PB are stratified.Simulation: 48 subjects, 2 stratification factors, 6 strata, uneven sizes

(DA) Dynamic Allocation (PB) Permuted Block (CR) Completely RandomDA( 2nd Best Probability ), PB( Allocation Ratio )

Simulated subjects were randomized by all 3 methods


Randomizations Plotted by Metrics

(Essentially Strict Minimization)

CR

PB(4:4)

DA(0.33)

PB(1:1), DA(0)

PB(2:2), DA(0.15)

PB(8:8)

DA(0.5)

CR

DA(0.5) CR ≣PB CR⟶


Correlation of Metrics


Simulated Comparison

• 1,000 simulations per case

* 48 subjects each

* 6 Strata, 2 factor, Variety of proportions

DA(0.25)PB(3:3)


Simulated Comparison

DA(0.25)PB(3:3)


Relative Loss of Efficiency

•

DA(0.25)PB(3:3)


Special TopicsLocal

Predictability

ONLY


Dynamic Allocation Weighting

DA(0) balanced only within strata Approximates PB(1:1)DA(0) equal weighting Approximates PB(1:1)DA(0) balanced on margins Intermediate propertiesDA(0) balanced only overall Approximates CR (large N)

NB: Predictability is limited to imbalance within a stratum


ONLY


Dynamic Allocation Weighting

Weighting: (Strata, Margins, Overall)

DA(0) Equal Weighting (1,1,1) Strata Balance Dominates Approximates PB(1:1)

DA(0) Margin & Strata (1:9:0) Separates from PB(1:1)

DA(0) Unequal Weighting (1,6,20)

DA(0) Margin Balance (0,1,0)

DA(0) Overall Balance (0,0,1) Approx. CR


ONLY


DA Algorithm

Distance function ≣ Weighted Sum of Imbalances

• Relative Imbalance:

• Factor as Union of Strata ⇒

⇒ Strata Imbalances dominate Distance function


Weighting

Over all Ages:

Pla

Test

Pla

Test

Pla Test

Over both sexes

Pla

Test

Pla

Test

Pla

Test

Pla

Test

Pla Test

Males Females

18-35 yo

35-65 yo

>65 yo

Pla Test

Pla

Test

Pla

Test

Pla Test

Over all Ages:

Pla

Test

Pla

Test

Pla Test

Over both sexes

Pla

Test

Pla

Test

Pla

Test

Pla

Test

Pla Test

Males Females

18-35 yo

35-65 yo

>65 yo

Pla Test

Pla

Test

Pla

Test

Pla Test

• Stratified Randomization weights on strata, not margins or overall

• Imbalances within strata tend to dominate in DA

• Minimization weights on margins, not strata.

• DA can weight exclusively on margins

Over all Ages:

Pla

Test

Pla

Test

Pla Test

Over both sexes

Pla

Test

Pla

Test

Pla

Test

Pla

Test

Pla Test

Males Females

18-35 yo

35-65 yo

>65 yo

Pla Test

Pla

Test

Pla

Test

Pla Test

• If a Strata is balanced, the next assignment attempts to balance the margins.

• Since small groups are more likely to have imbalances which reduce efficiency, balancing strata 1st is appropriate


Hierarchical Balancing

Over all Ages:

PlaTest

Pla

Test

Pla Test

Over both sexes

PlaTest

PlaTest

PlaTest

PlaTest

PlaTest

Males Females

18-35 yo

35-65 yo

>65 yo

PlaTest

PlaTest

PlaTest

Pla Test

• While Imbalances within strata tends to dominate in DA,if a Strata is balanced, the next assignment attempts to balance the margins

• Since small group imbalances tend to dominate, balancing tends to be sequential

⟵ This example:(1) Balance within strata (2) If balanced within the strata, balance by age group

(since age groups tend to be smaller than sex groups)(3) If balanced within age group, balance within sex group(4) If balanced within sex group, balance overall

However: cumulative imbalances may change this order


Replacement Randomization


Dynamically Adapting to Dropouts

Patients discontinue ⟶ Imbalances ⟶ Reduced efficiency

“Tight” randomizations(PB with small blocks, DA with small 2nd best Prob.)

⟶ Lose more efficiency

“Loose” randomizations(CR, PB with large blocks, DA with large 2nd best Prob.)

⟶ Lose less efficiency ⟶ Little or no change

No DC

25% DC



Dynamic Allocation: Can allocate new patients to restore balance

No DC

25% DC

DA Adj.



“Tight” randomizations(PB with small blocks, DA with small 2nd best Prob.)

⟶ Lose more efficiency ⟶ Benefit most

“Loose” randomizations(CR, PB with large blocks, DA with large 2nd best Prob.)

⟶ Lose less efficiency ⟶ Little or no benefit

No DC

25% DC

DA Adj.


Applications

• High drop-out ⇒ PB, DA ⟶ CR• Drop-out before becoming evaluable• Constrained resources (small sample size, limited drug supply, ….)

• Crossover studies: Requires completers• Evaluable Complete Sequence of Treatments

• Provisional Randomization / Randomize to ship• Screening visit triggers:

• Randomize at screening• If randomized treatment not on-site, ship blinded supplies

• Randomization visit:• If patient eligible ⇒ dispense assigned treatment• If not eligible ⇒store for next eligible patient

• Minimizes on-site drug supply


Randomization Optimization Factors

• Equipose ⇒ (less random is acceptable)• Small Study ⇒ Efficiency important

⟶ Lower 2nd Best Probability• Large Study ⇒ Are there small subgroups?

All subgroups large ⟶ CR is acceptable• Small subgroups ⇒ Need more efficiency

⟶ Smaller 2nd best Prob


Balancing Considerations

Unpredictable ⟵ ⟵⟵ ⟶ ⟶ ⟶ Balanced

• Smaller Studies• Studies with small

subgroups• Early phase studies• Interim Analyses• Equipoise • Strong Blinding• Objective Endpoints• Many Strata / Many centers• Limited blinded supplies

• Large Studies• Studies with large subgroups• Late phase studies• Strong Treatment preferences• Weak Blinding• Subjective Endpoints


Bibliography

Elsa Valdes Márquez & Nick Fieller. Inference in Covariate-Adaptive allocation. EFSPI Adaptive Randomisation Meeting, Brussels, 7 December 2006

Randomization: Too Important to Gamble with

Documents

Transcript of Randomization: Too Important to Gamble with