Experimental design and sample size determination Karl W Broman Department of Biostatistics Johns...
-
Upload
nickolas-reynolds -
Category
Documents
-
view
218 -
download
0
Transcript of Experimental design and sample size determination Karl W Broman Department of Biostatistics Johns...
![Page 1: Experimental design and sample size determination Karl W Broman Department of Biostatistics Johns Hopkins University kbroman.](https://reader035.fdocuments.net/reader035/viewer/2022062716/56649dc05503460f94ab48c7/html5/thumbnails/1.jpg)
Experimental designand sample size determination
Karl W Broman
Department of Biostatistics
Johns Hopkins University
http://www.biostat.jhsph.edu/~kbroman
![Page 2: Experimental design and sample size determination Karl W Broman Department of Biostatistics Johns Hopkins University kbroman.](https://reader035.fdocuments.net/reader035/viewer/2022062716/56649dc05503460f94ab48c7/html5/thumbnails/2.jpg)
Note
• This is a shortened version of a lecture which is part of a web-based course on “Enhancing Humane Science/Improving Animal Research” (organized by Alan Goldberg, Johns Hopkins Center for Alternatives to Animal Testing)
• Few details—mostly concepts.
2
![Page 3: Experimental design and sample size determination Karl W Broman Department of Biostatistics Johns Hopkins University kbroman.](https://reader035.fdocuments.net/reader035/viewer/2022062716/56649dc05503460f94ab48c7/html5/thumbnails/3.jpg)
Experimental design
3
![Page 4: Experimental design and sample size determination Karl W Broman Department of Biostatistics Johns Hopkins University kbroman.](https://reader035.fdocuments.net/reader035/viewer/2022062716/56649dc05503460f94ab48c7/html5/thumbnails/4.jpg)
Basic principles
1. Formulate question/goal in advance
2. Comparison/control
3. Replication
4. Randomization
5. Stratification (aka blocking)
6. Factorial experiments
4
![Page 5: Experimental design and sample size determination Karl W Broman Department of Biostatistics Johns Hopkins University kbroman.](https://reader035.fdocuments.net/reader035/viewer/2022062716/56649dc05503460f94ab48c7/html5/thumbnails/5.jpg)
Example
Question: Does salted drinking water affect blood pressure (BP) in mice?
Experiment:
1. Provide a mouse with water containing 1% NaCl.
2. Wait 14 days.
3. Measure BP.
5
![Page 6: Experimental design and sample size determination Karl W Broman Department of Biostatistics Johns Hopkins University kbroman.](https://reader035.fdocuments.net/reader035/viewer/2022062716/56649dc05503460f94ab48c7/html5/thumbnails/6.jpg)
Comparison/control
Good experiments are comparative.
• Compare BP in mice fed salt water to BP in mice fed plain water.
• Compare BP in strain A mice fed salt water to BP in strain B mice fed salt water.
Ideally, the experimental group is compared to concurrent controls (rather than to historical controls).
6
![Page 7: Experimental design and sample size determination Karl W Broman Department of Biostatistics Johns Hopkins University kbroman.](https://reader035.fdocuments.net/reader035/viewer/2022062716/56649dc05503460f94ab48c7/html5/thumbnails/7.jpg)
Replication
7
![Page 8: Experimental design and sample size determination Karl W Broman Department of Biostatistics Johns Hopkins University kbroman.](https://reader035.fdocuments.net/reader035/viewer/2022062716/56649dc05503460f94ab48c7/html5/thumbnails/8.jpg)
Why replicate?
• Reduce the effect of uncontrolled variation (i.e., increase precision).
• Quantify uncertainty.
A related point:
An estimate is of no value without some statement of the uncertainty in the estimate.
8
![Page 9: Experimental design and sample size determination Karl W Broman Department of Biostatistics Johns Hopkins University kbroman.](https://reader035.fdocuments.net/reader035/viewer/2022062716/56649dc05503460f94ab48c7/html5/thumbnails/9.jpg)
Randomization
Experimental subjects (“units”) should be assigned to treatment groups at random.
At random does not mean haphazardly.
One needs to explicitly randomize using• A computer, or• Coins, dice or cards.
9
![Page 10: Experimental design and sample size determination Karl W Broman Department of Biostatistics Johns Hopkins University kbroman.](https://reader035.fdocuments.net/reader035/viewer/2022062716/56649dc05503460f94ab48c7/html5/thumbnails/10.jpg)
Why randomize?
• Avoid bias.– For example: the first six mice you grab may have intrinsicly
higher BP.
• Control the role of chance.– Randomization allows the later use of probability theory, and
so gives a solid foundation for statistical analysis.
10
![Page 11: Experimental design and sample size determination Karl W Broman Department of Biostatistics Johns Hopkins University kbroman.](https://reader035.fdocuments.net/reader035/viewer/2022062716/56649dc05503460f94ab48c7/html5/thumbnails/11.jpg)
Stratification
• Suppose that some BP measurements will be made in the morning and some in the afternoon.
• If you anticipate a difference between morning and afternoon measurements:– Ensure that within each period, there are equal numbers of
subjects in each treatment group.– Take account of the difference between periods in your
analysis.
• This is sometimes called “blocking”.
11
![Page 12: Experimental design and sample size determination Karl W Broman Department of Biostatistics Johns Hopkins University kbroman.](https://reader035.fdocuments.net/reader035/viewer/2022062716/56649dc05503460f94ab48c7/html5/thumbnails/12.jpg)
Example
• 20 male mice and 20 female mice.
• Half to be treated; the other half left untreated.
• Can only work with 4 mice per day.
Question: How to assign individuals to treatmentgroups and to days?
12
![Page 13: Experimental design and sample size determination Karl W Broman Department of Biostatistics Johns Hopkins University kbroman.](https://reader035.fdocuments.net/reader035/viewer/2022062716/56649dc05503460f94ab48c7/html5/thumbnails/13.jpg)
An extremelybad design
13
![Page 14: Experimental design and sample size determination Karl W Broman Department of Biostatistics Johns Hopkins University kbroman.](https://reader035.fdocuments.net/reader035/viewer/2022062716/56649dc05503460f94ab48c7/html5/thumbnails/14.jpg)
Randomized
14
![Page 15: Experimental design and sample size determination Karl W Broman Department of Biostatistics Johns Hopkins University kbroman.](https://reader035.fdocuments.net/reader035/viewer/2022062716/56649dc05503460f94ab48c7/html5/thumbnails/15.jpg)
A stratified design
15
![Page 16: Experimental design and sample size determination Karl W Broman Department of Biostatistics Johns Hopkins University kbroman.](https://reader035.fdocuments.net/reader035/viewer/2022062716/56649dc05503460f94ab48c7/html5/thumbnails/16.jpg)
Randomization and
stratification• If you can (and want to), fix a variable.
– e.g., use only 8 week old male mice from a single strain.
• If you don’t fix a variable, stratify it.– e.g., use both 8 week and 12 week old male mice, and stratify
with respect to age.
• If you can neither fix nor stratify a variable, randomize it.
16
![Page 17: Experimental design and sample size determination Karl W Broman Department of Biostatistics Johns Hopkins University kbroman.](https://reader035.fdocuments.net/reader035/viewer/2022062716/56649dc05503460f94ab48c7/html5/thumbnails/17.jpg)
Factorial experiments
Suppose we are interested in the effect of both salt water and a high-fat diet on blood pressure.
Ideally: look at all 4 treatments in one experiment.
Plain water Normal dietSalt water High-fat diet
Why?– We can learn more.– More efficient than doing all single-factor
experiments.
17
![Page 18: Experimental design and sample size determination Karl W Broman Department of Biostatistics Johns Hopkins University kbroman.](https://reader035.fdocuments.net/reader035/viewer/2022062716/56649dc05503460f94ab48c7/html5/thumbnails/18.jpg)
Interactions
18
![Page 19: Experimental design and sample size determination Karl W Broman Department of Biostatistics Johns Hopkins University kbroman.](https://reader035.fdocuments.net/reader035/viewer/2022062716/56649dc05503460f94ab48c7/html5/thumbnails/19.jpg)
Other points
• Blinding– Measurements made by people can be influenced by
unconscious biases.– Ideally, dissections and measurements should be made
without knowledge of the treatment applied.
• Internal controls– It can be useful to use the subjects themselves as their own
controls (e.g., consider the response after vs. before treatment).
– Why? Increased precision.
19
![Page 20: Experimental design and sample size determination Karl W Broman Department of Biostatistics Johns Hopkins University kbroman.](https://reader035.fdocuments.net/reader035/viewer/2022062716/56649dc05503460f94ab48c7/html5/thumbnails/20.jpg)
Other points
• Representativeness– Are the subjects/tissues you are studying really
representative of the population you want to study?– Ideally, your study material is a random sample from the
population of interest.
20
![Page 21: Experimental design and sample size determination Karl W Broman Department of Biostatistics Johns Hopkins University kbroman.](https://reader035.fdocuments.net/reader035/viewer/2022062716/56649dc05503460f94ab48c7/html5/thumbnails/21.jpg)
Summary
• Unbiased– Randomization
– Blinding
• High precision– Uniform material
– Replication
– Blocking
• Simple– Protect against mistakes
• Wide range of applicability– Deliberate variation
– Factorial designs
• Able to estimate uncertainty– Replication
– Randomization
Characteristics of good experiments:
21
![Page 22: Experimental design and sample size determination Karl W Broman Department of Biostatistics Johns Hopkins University kbroman.](https://reader035.fdocuments.net/reader035/viewer/2022062716/56649dc05503460f94ab48c7/html5/thumbnails/22.jpg)
Data presentation
0
5
10
15
20
25
30
35
40
A B
Group
Bad plotGood plot
22
![Page 23: Experimental design and sample size determination Karl W Broman Department of Biostatistics Johns Hopkins University kbroman.](https://reader035.fdocuments.net/reader035/viewer/2022062716/56649dc05503460f94ab48c7/html5/thumbnails/23.jpg)
Data presentation
Treatment Mean (SEM)
A 11.2 (0.6)
B 13.4 (0.8)
C 14.7 (0.6)
Treatment Mean (SEM)
A 11.2965 (0.63)
B 13.49 (0.7913)
C 14.787 (0.6108)
Good table Bad table
23
![Page 24: Experimental design and sample size determination Karl W Broman Department of Biostatistics Johns Hopkins University kbroman.](https://reader035.fdocuments.net/reader035/viewer/2022062716/56649dc05503460f94ab48c7/html5/thumbnails/24.jpg)
Sample size determination
24
![Page 25: Experimental design and sample size determination Karl W Broman Department of Biostatistics Johns Hopkins University kbroman.](https://reader035.fdocuments.net/reader035/viewer/2022062716/56649dc05503460f94ab48c7/html5/thumbnails/25.jpg)
Fundamental formula
sample per $available $
n =
25
![Page 26: Experimental design and sample size determination Karl W Broman Department of Biostatistics Johns Hopkins University kbroman.](https://reader035.fdocuments.net/reader035/viewer/2022062716/56649dc05503460f94ab48c7/html5/thumbnails/26.jpg)
Listen to the IACUC
Too few animals a total waste
Too many animals a partial waste
26
![Page 27: Experimental design and sample size determination Karl W Broman Department of Biostatistics Johns Hopkins University kbroman.](https://reader035.fdocuments.net/reader035/viewer/2022062716/56649dc05503460f94ab48c7/html5/thumbnails/27.jpg)
Significance test
• Compare the BP of 6 mice fed salt water to 6 mice fed plain water.
= true difference in average BP (the treatment effect).
• H0: = 0 (i.e., no effect)• Test statistic, D.• If |D| > C, reject H0.• C chosen so that the chance
you reject H0, if H0 is true, is 5%
Distribution of D when = 0
27
![Page 28: Experimental design and sample size determination Karl W Broman Department of Biostatistics Johns Hopkins University kbroman.](https://reader035.fdocuments.net/reader035/viewer/2022062716/56649dc05503460f94ab48c7/html5/thumbnails/28.jpg)
Statistical power
Power = The chance that you reject H0 when H0 is false (i.e., you [correctly] conclude that there is a treatment effect when there really is a treatment effect).
28
![Page 29: Experimental design and sample size determination Karl W Broman Department of Biostatistics Johns Hopkins University kbroman.](https://reader035.fdocuments.net/reader035/viewer/2022062716/56649dc05503460f94ab48c7/html5/thumbnails/29.jpg)
Power depends on…
• The structure of the experiment• The method for analyzing the data• The size of the true underlying effect• The variability in the measurements• The chosen significance level ()• The sample size
Note: We usually try to determine the sample size to give a particular power (often 80%).
29
![Page 30: Experimental design and sample size determination Karl W Broman Department of Biostatistics Johns Hopkins University kbroman.](https://reader035.fdocuments.net/reader035/viewer/2022062716/56649dc05503460f94ab48c7/html5/thumbnails/30.jpg)
Effect of sample size
6 per group:
12 per group:
30
![Page 31: Experimental design and sample size determination Karl W Broman Department of Biostatistics Johns Hopkins University kbroman.](https://reader035.fdocuments.net/reader035/viewer/2022062716/56649dc05503460f94ab48c7/html5/thumbnails/31.jpg)
Effect of the effect
= 8.5:
= 12.5:
31
![Page 32: Experimental design and sample size determination Karl W Broman Department of Biostatistics Johns Hopkins University kbroman.](https://reader035.fdocuments.net/reader035/viewer/2022062716/56649dc05503460f94ab48c7/html5/thumbnails/32.jpg)
Various effects
• Desired power sample size
• Stringency of statistical test sample size
• Measurement variability sample size
• Treatment effect sample size
32
![Page 33: Experimental design and sample size determination Karl W Broman Department of Biostatistics Johns Hopkins University kbroman.](https://reader035.fdocuments.net/reader035/viewer/2022062716/56649dc05503460f94ab48c7/html5/thumbnails/33.jpg)
Determining sample size
The things you need to know:
• Structure of the experiment• Method for analysis• Chosen significance level, (usually 5%)• Desired power (usually 80%)
• Variability in the measurements – if necessary, perform a pilot study
• The smallest meaningful effect
33
![Page 34: Experimental design and sample size determination Karl W Broman Department of Biostatistics Johns Hopkins University kbroman.](https://reader035.fdocuments.net/reader035/viewer/2022062716/56649dc05503460f94ab48c7/html5/thumbnails/34.jpg)
A formula
[ ] 2n 212/
2×−⎟
⎠⎞
⎜⎝⎛
Δ= −βα
σzzCensored
34
![Page 35: Experimental design and sample size determination Karl W Broman Department of Biostatistics Johns Hopkins University kbroman.](https://reader035.fdocuments.net/reader035/viewer/2022062716/56649dc05503460f94ab48c7/html5/thumbnails/35.jpg)
Reducing sample size
• Reduce the number of treatment groups being compared.
• Find a more precise measurement (e.g., average time to effect rather than proportion sick).
• Decrease the variability in the measurements.– Make subjects more homogeneous.– Use stratification.– Control for other variables (e.g., weight).– Average multiple measurements on each subject.
35
![Page 36: Experimental design and sample size determination Karl W Broman Department of Biostatistics Johns Hopkins University kbroman.](https://reader035.fdocuments.net/reader035/viewer/2022062716/56649dc05503460f94ab48c7/html5/thumbnails/36.jpg)
Final conclusions
• Experiments should be designed.
• Good design and good analysis can lead to reduced sample sizes.
• Consult an expert on both the analysis and the design of your experiment.
36
![Page 37: Experimental design and sample size determination Karl W Broman Department of Biostatistics Johns Hopkins University kbroman.](https://reader035.fdocuments.net/reader035/viewer/2022062716/56649dc05503460f94ab48c7/html5/thumbnails/37.jpg)
Resources
• ML Samuels, JA Witmer (2003) Statistics for the Life Sciences, 3rd edition. Prentice Hall.– An excellent introductory text.
• GW Oehlert (2000) A First Course in Design and Analysis of Experiments. WH Freeman & Co.– Includes a more advanced treatment of experimental design.
• Course: Statistics for Laboratory Scientists (Biostatistics 140.615-616, Johns Hopkins Bloomberg Sch. Pub. Health)– Intoductory statistics course, intended for experimental
scientists.– Greatly expands upon the topics presented here.
37