Post on 01-Jan-2016
description
Sample Size Considerations for Answering Quantitative
Research Questions
Lunch & Learn May 15, 2013 M Boyle
Fiscal Year 2007
68.8
Fiscal Year 2008
110.9
Fiscal Year 2009
179.8
Fiscal Year 2010
193.9
Fiscal Year 2011
191.1
Sum
744.6
National Children’s Study in the US
Proposed Birth Cohort 100,000 to age 21Planning Costs 2000-2006: $54.7M
Implementation Costs 2007-2011: $744.6
Sample Size Justification: ?
What is Statistical Power?
• The statistical power of a test is the probability of correctly rejecting H0 when it is false. In other words, power is the likelihood that you will identify a statistically significant effect when one exists
Types of Power Analysis
A priori: Used to plan a study (Question: What sample size is needed to obtain a certain level of power)?
Post hoc: Used to evaluate a study faced with a constrained sample size (Question: Do you have a large enough sample to detect a meaningful effect)?
[Types of constraints: (1) a completed study; (2) a proposed study with limited number of eligible subjects; (3) a proposed study faced with limited resources]
Elements of Power Calculations
• Effect size ∆• Measurement variability SD • Type I error Alpha (α)
typically specified at p=0.05, 2-tailed • Type II error Beta (β) typically specified at p=0.20 • Power = 1-β; typically 0.80 • Sample Size
Hypothesized distributions, effect sizes and error rates
0
0.1
0.2
0.3
-3 -2 -1 0 1 2 3 4 5
critical t = 1.97897
α2β
Effect Size ∆
Type IType II
Measurement Variability +/- 1 SD
Decisions
Medical Diagnosis
Hypothesis Testing
Disease StatusPresent Absent
Population Status H0 false
H0 true [H1 true]
+ve
Test Result
-ve
Accept H0
Decision
Reject H0
[Accept H1]
correct false +ve
false -ve correct
correct Type II 1-α β
Type I correct α 1-β (power)
Example Power Calculation
H0: At 2 years of age, the IQs of newborns randomly allocated to the NFP program will be no different than newborns allocated to usual care.
H1: At 2 years of age, the IQs of newborns randomly allocated the NFP program will be 5 points higher.
Effect size ∆ =
SD =
Alpha (α) =
Beta (β) =
Power =
Sample Size ?
Example Power Calculation
H0: At 2 years of age, the IQs of newborns randomly allocated to the NFP program will be no different than newborns allocated to usual care.
H1: At 2 years of age, the IQs of newborns randomly allocated the NFP program will be 5 points higher.
Effect size ∆ = 5
SD = 15
Alpha (α) = 0.05 2-tailed
Beta (β) = 0.20
Power = 80
Sample Size 146 per group
Effect size d
Tota
l sam
ple
siz
e
t tests - Means: Difference between two independent means (two groups)Tail(s) = Two, Allocation ratio N2/N1 = 1,
α err prob = 0.05, Power (1-β err prob) = 0.8
0
500
1000
1500
2000
2500
3000
0.1 0.15 0.2 0.25 0.3 0.35 0.4 0.45 0.5
n=292
Sample Size Planning and Power
1. Error rates Type I (α) -smaller α requires larger sample sizes -2-sided tests requires larger sample sizes
Type II (β) Statistical power: -smaller β (more power) requires larger sample sizes
[Use conventional levels & worry about the trade- offs between effect size and sample size]
Sample Size Planning and Power
2. Effect Size ∆
“What is the minimally important effect based on clinical, biological or social implications of the findings?”
Sample Size Planning and Power
1. Effect size ∆
• What do you know about the nature of the effect – its scale of measurement and its perceived importance to practice, policy, resource allocation (e.g., infant mortality; dollars; self-esteem)?
• What do previous empirical studies tell you about achievable effects?
Sample Size Planning and Power
1. Effect Size ∆
• Can you generate a consensus among your investigative team on a minimally important effect?
• Is it reasonable to use conventional estimates of small, medium and large?
• Are you limited by the dollar amount you can request?
Sample Size Planning and Power
2. The measurement scale of the dependent variable: discrete, ordinal, interval
-interval level measurements require smaller samples3. The variability of the dependent variable
in the general population (SD, Variance) -lower variability requires smaller sample sizes
Sample Size Planning and Power
4. The statistical test -simple estimation; differences between
groups; correlation and prediction. The test must be appropriate for the question and data. A key element in sample size planning
5. Sample distribution, for example, exposed versus not exposed)
-balanced is the most powerful
Sample Size Planning and Power
6. Attrition loss of subjects -higher attrition leads to lower power
7. Measurement reliability -complicated: if true variance is constant
and error variance is reduced statistical power will increase
Sample Size Planning and Power
8. Study costs – what the market will bear
9. Analytical complexity – what to do when your models require much more information than you can get?
Adding Complexity
• Multilevel Model yij = β0j + β1z0j + (u0j + eij)
H0 The association between neighbourhood affluence measured on resident 4-16 year olds in 1983 and years of education assessed in 2001 will be = 0.00 standard units
Neigh Affluence
x
y
H0 ∆ = β1z0j > 0.20
Estimates
• 2-level balanced data, nested model• Significance level = 0.025 (to get 0.05 2-
tailed)• Number of simulations per setting = 100• Response variable = normal• Estimation method = IGLS• Fixed intercept = yes• Random intercept = yes• Number of explanatory variables = 1• Type of predictor = continuous
Estimates
• Mean of the predictor = 0.0• Variance of the predictor at level 1 = 0.0• Variance of the predictor at level 2 = 1.0• Smallest/Largest # units at L1 (increment)• Smallest/Largest # units at L2 (increment)• Estimate β0 = 0• Estimate β1 = 0.15• Estimate L2 variance 0.05• Estimate L1 variance 0.95
Comments• Ask specific, quantifiable research questions • Consult with colleagues about clinical,
biological and social importance of your outcomes
• Move from simple to complex hypotheses. Complex models – SEM, Multilevel – can require you to provide an enormous number of parameters.
• When estimating sample size requirements for complex models, you will inevitably use standardized variables
Comments• Estimating sample size requirements is part
game, subject to practical constraints (limited resources and subjects) and convincing reviewers that you know what your doing
• Take a ‘reasoned’ approach – most reviewers will have no clue what you are going on about
• The hardest part of the process is acquiring the information you need.