Consequences of Sequential Sampling for Meta-analysis

Consequences ofSequential

Sampling forMeta-Analysis

Lorenzo BraschiDiaferia, Juan

Botella Ausina,Manuel Suero

Introduction

What are sequentialsampling rules?

The CLAST rule

The CLAST rule andmeta-analysis

Method

Monte Carlo Simulations

Simulation parameters

Results

Sample size

Effect size estimation andbias

Weighting

Conclusions

Consequences of Sequential Sampling forMeta-Analysis

Lorenzo Braschi Diaferia 1 Juan Botella Ausina 2

Manuel Suero Sune 2

1Facultad de Ciencias de la SaludUniversidad Alfonso X el Sabio

2Facultad de PsicologıaUniversidad Autonoma de Madrid

July 20th

Introduction

The CLAST rule

Method

Results

Sample size

Weighting

Conclusions

Varying sample size

I Classical hypothesistesting requires a fixedsampling procedure to setα and β levels.

I But being able to adjustsample size as a result ofanalysis is cost-effective.

I Sequential sampling ruleshave been developed forseveral applications.

Introduction

The CLAST rule

Method

Results

Sample size

Weighting

Conclusions

Varying sample size

Introduction

The CLAST rule

Method

Results

Sample size

Weighting

Conclusions

Varying sample size

Introduction

The CLAST rule

Method

Results

Sample size

Weighting

Conclusions

What are sequential sampling rules?

A sequential sampling rule is a procedure for hypothesistesting that uses a smaller sample size relative to astandard or fixed sampling procedure.

The purpose of a sequential sampling rule is:

1. To offer a more flexible procedure for hypothesis testingregarding sample size.

2. To keep type I and type II error probabilities α and β

under control.

Introduction

The CLAST rule

Method

Results

Sample size

Weighting

Conclusions

under control.

Introduction

The CLAST rule

Method

Results

Sample size

Weighting

Conclusions

under control.

Introduction

The CLAST rule

Method

Results

Sample size

Weighting

Conclusions

The CLAST rule

Composite Limited Adaptive Sequential Test (Botella,Ximenez, Revuelta, Suero, 2006)

I Set a lower and an upper boundary on sample sizeI Divide region under the statistic’s distribution in three

different areas:

1. A rejection area2. A manteinance area3. An uncertainty area that demands an increase in sample

I Calculate test statistic and associate p value.

I If p value falls within the uncertainty area, add moresubjects to the sample.

I Repeat until stopping conditions are met.

Introduction

The CLAST rule

Method

Results

Sample size

Weighting

Conclusions

The CLAST rule

I Set a lower and an upper boundary on sample size

I Divide region under the statistic’s distribution in threedifferent areas:

Introduction

The CLAST rule

Method

Results

Sample size

Weighting

Conclusions

The CLAST rule

different areas:

Introduction

The CLAST rule

Method

Results

Sample size

Weighting

Conclusions

The CLAST rule

different areas:

1. A rejection area

2. A manteinance area3. An uncertainty area that demands an increase in sample

Introduction

The CLAST rule

Method

Results

Sample size

Weighting

Conclusions

The CLAST rule

different areas:

1. A rejection area2. A manteinance area

3. An uncertainty area that demands an increase in samplesize

Introduction

The CLAST rule

Method

Results

Sample size

Weighting

Conclusions

The CLAST rule

different areas:

Introduction

The CLAST rule

Method

Results

Sample size

Weighting

Conclusions

The CLAST rule

different areas:

Introduction

The CLAST rule

Method

Results

Sample size

Weighting

Conclusions

The CLAST rule

different areas:

Introduction

The CLAST rule

Method

Results

Sample size

Weighting

Conclusions

The CLAST rule

different areas:

Introduction

The CLAST rule

Method

Results

Sample size

Weighting

Conclusions

The CLAST rule

Introduction

The CLAST rule

Method

Results

Sample size

Weighting

Conclusions

CLAST parameters and variables

Parameters:

1. Reference sample size for equivalent fixed sample size:NFSR .

2. Initial starting sample size: N1.

3. Upper and lower boundaries for the uncertainty areas:α1 and αu.

Variables:

1. Sample size at stopping: NSTOP

Introduction

The CLAST rule

Method

Results

Sample size

Weighting

Conclusions

Parameters:

Variables:

Introduction

The CLAST rule

Method

Results

Sample size

Weighting

Conclusions

Parameters:

Variables:

Introduction

The CLAST rule

Method

Results

Sample size

Weighting

Conclusions

Parameters:

Variables:

Introduction

The CLAST rule

Method

Results

Sample size

Weighting

Conclusions

Parameters:

Variables:

Introduction

The CLAST rule

Method

Results

Sample size

Weighting

Conclusions

The CLAST rule for Student’s paired t test

Botella et al. (2006) found that the CLAST rule uses asample size 20-40% smaller than a fixed rule of equivalentpower.

Introduction

The CLAST rule

Method

Results

Sample size

Weighting

Conclusions

The CLAST rule and meta-analysis

Main problem: How to incorporate primary CLAST studiesinto a meta-analysis?

We need to know:

1. Bias of estimations of effect size under CLAST.

2. Appropiate weights for primary CLAST studies.

Introduction

The CLAST rule

Method

Results

Sample size

Weighting

Conclusions

We need to know:

Introduction

The CLAST rule

Method

Results

Sample size

Weighting

Conclusions

We need to know:

Introduction

The CLAST rule

Method

Results

Sample size

Weighting

Conclusions

I Monte Carlo simulations of random samples using R.

I Student’s paired t test.

I Population parameters: µ and σ (so that the status ofthe null H0 is TRUE/FALSE).

I For all studies, µ = δ.

I Sample of initial size N1 drawn from theoreticalpopulation of differences D ∼ N(δ,1).

I New data added drawn from the same population.

Introduction

The CLAST rule

Method

Results

Sample size

Weighting

Conclusions

Introduction

The CLAST rule

Method

Results

Sample size

Weighting

Conclusions

Introduction

The CLAST rule

Method

Results

Sample size

Weighting

Conclusions

Introduction

The CLAST rule

Method

Results

Sample size

Weighting

Conclusions

Introduction

The CLAST rule

Method

Results

Sample size

Weighting

Conclusions

Introduction

The CLAST rule

Method

Results

Sample size

Weighting

Conclusions

Introduction

The CLAST rule

Method

Results

Sample size

Weighting

Conclusions

I N fixed= {16, 20, 24, 30, 40}I Theoretical effect size δ = {0, 0.1, 0.2, 0.3, ... , 1}

I If δ = 0, then H0 = TRUEI If δ > 0, then H0 = FALSE

I 105 iterations for each condition (combination of Nfixed and δ, 55 total scenarios)

I α1 = 0.01

I αu = 0.25

I α1 and αu values have been selected so that the overallα level converges to α = 0.05 in the long run (afterBotella et al., 2006)

Introduction

The CLAST rule

Method

Results

Sample size

Weighting

Conclusions

I N fixed= {16, 20, 24, 30, 40}

I Theoretical effect size δ = {0, 0.1, 0.2, 0.3, ... , 1}

I α1 = 0.01

I αu = 0.25

Introduction

The CLAST rule

Method

Results

Sample size

Weighting

Conclusions

I α1 = 0.01

I αu = 0.25

Introduction

The CLAST rule

Method

Results

Sample size

Weighting

Conclusions

I If δ = 0, then H0 = TRUE

I If δ > 0, then H0 = FALSE

I α1 = 0.01

I αu = 0.25

Introduction

The CLAST rule

Method

Results

Sample size

Weighting

Conclusions

I α1 = 0.01

I αu = 0.25

Introduction

The CLAST rule

Method

Results

Sample size

Weighting

Conclusions

I α1 = 0.01

I αu = 0.25

Introduction

The CLAST rule

Method

Results

Sample size

Weighting

Conclusions

I α1 = 0.01

I αu = 0.25

Introduction

The CLAST rule

Method

Results

Sample size

Weighting

Conclusions

I α1 = 0.01

I αu = 0.25

Introduction

The CLAST rule

Method

Results

Sample size

Weighting

Conclusions

Results

Introduction

The CLAST rule

Method

Results

Sample size

Weighting

Conclusions

CLAST rule mean sample size at stopping fordifferent starting effect sizes

Introduction

The CLAST rule

Method

Results

Sample size

Weighting

Conclusions

Effect size estimation and bias under CLAST

We used the d as an estimator of δ for the paired t test:

SD· c(m) (1)

Bias was measured as the difference between the estimatorand the true value of the effect size:

Bias = d−δ (2)

Introduction

The CLAST rule

Method

Results

Sample size

Weighting

Conclusions

Estimation of δ under the CLAST and fixedsampling rules

Introduction

The CLAST rule

Method

Results

Sample size

Weighting

Conclusions

Effect size estimation and bias under CLAST

No significant bias for effect size estimation.

Primary CLAST studies can be incorporated into ameta-analysis without adding significant bias.

Introduction

The CLAST rule

Method

Results

Sample size

Weighting

Conclusions

How to weight primary CLAST studies?

Common procedure: weight studies by inverse of variance ofestimator d .

We need to know var(d)

var(d) =∑wi ·di

∑wi(3)

Where wi = N

Introduction

The CLAST rule

Method

Results

Sample size

Weighting

Conclusions

How to calculate w?

To calculate var(d), we need to weight studies by N, that is,sample size.

Which sample size?

In CLAST, we have three possible candidates for sample sizeweighting:

1. Equivalent sample size under the fixed sampling rule,NFSR .

2. Starting sample size, N1.

3. Sample size at stopping, NSTOP

Introduction

The CLAST rule

Method

Results

Sample size

Weighting

Conclusions

How to calculate w?

Which sample size?

Introduction

The CLAST rule

Method

Results

Sample size

Weighting

Conclusions

How to calculate w?

Which sample size?

Introduction

The CLAST rule

Method

Results

Sample size

Weighting

Conclusions

How to calculate w?

Which sample size?

Introduction

The CLAST rule

Method

Results

Sample size

Weighting

Conclusions

How to calculate w?

Which sample size?

Introduction

The CLAST rule

Method

Results

Sample size

Weighting

Conclusions

How to calculate w?

Which sample size?

Introduction

The CLAST rule

Method

Results

Sample size

Weighting

Conclusions

Mean CLAST weights for NFSR = 24

Introduction

The CLAST rule

Method

Results

Sample size

Weighting

Conclusions

Introduction

The CLAST rule

Method

Results

Sample size

Weighting

Conclusions

Introduction

The CLAST rule

Method

Results

Sample size

Weighting

Conclusions

Introduction

The CLAST rule

Method

Results

Sample size

Weighting

Conclusions

I No previous research on how to incorporate sequentialsampling rules into meta-analysis.

I Small to insignificant bias in effect size estimation underCLAST.

I Problem of weighting only when we mix CLAST andstandard primary studies.

I Use mean of NFSR and initial N.

Introduction

The CLAST rule

Method

Results

Sample size

Weighting

Conclusions

Introduction

The CLAST rule

Method

Results

Sample size

Weighting

Conclusions

Introduction

The CLAST rule

Method

Results

Sample size

Weighting

Conclusions

Introduction

The CLAST rule

Method

Results

Sample size

Weighting

Conclusions

Introduction

The CLAST rule

Method

Results

Sample size

Weighting

Conclusions

Thank you for yourattention!

Introduction

The CLAST rule

Method

Results

Sample size

Weighting

Conclusions

The CLAST rule algorithm for the paired t test

1. Determine the necessary sample under the fixed stopping rule(NFSR) as a function of desired power and the assumed effectsize.

2. Determine the initial sample size, N1 = 0.5NFSR , and themaximum sample size, Nmax = 1.5NFSR .

3. Execute the experiment with a sample of size N1 and analyzedata.

3.1 If p > αU , stop the experiment and maintain H0.3.2 If p ≤ α1, stop the experiment and reject H0.3.3 If α1 < p ≤ αU , add N2 = 1 to the sample.

4. If N1 +N2 = Nmax stop the experiment and decide:

4.1 If p ≤ 0.05 reject H0.4.2 If p > 0.05 maintain H0.

5. If N1 +N2 < Nmax reanalyze data with N1 +N2 and return tostep 3.

Introduction

The CLAST rule

Method

Results

Sample size

Weighting

Conclusions

Introduction

The CLAST rule

Method

Results

Sample size

Weighting

Conclusions

Introduction

The CLAST rule

Method

Results

Sample size

Weighting

Conclusions

Introduction

The CLAST rule

Method

Results

Sample size

Weighting

Conclusions

Introduction

The CLAST rule

Method

Results

Sample size

Weighting

Conclusions

Introduction

The CLAST rule

Method

Results

Sample size

Weighting

Conclusions

Introduction

The CLAST rule

Method

Results

Sample size

Weighting

Conclusions

Introduction

The CLAST rule

Method

Results

Sample size

Weighting

Conclusions

Introduction

The CLAST rule

Method

Results

Sample size

Weighting

Conclusions

Consequences of Sequential Sampling for Meta-analysis

Documents

Transcript of Consequences of Sequential Sampling for Meta-analysis