Business Journalism Professors 2014: Creating a CBP Lesson by Steve Doig
University of Sydney Statistics 101: Power, p-values and ………... publications. Dr. Gordon S...
-
date post
19-Dec-2015 -
Category
Documents
-
view
218 -
download
1
Transcript of University of Sydney Statistics 101: Power, p-values and ………... publications. Dr. Gordon S...
University of Sydney
Statistics 101: Power, p-values and………... publications.
Dr. Gordon S Doig,Senior Lecturer in Intensive Care,
Northern Clinical [email protected]
www.EvidenceBased.net/talks
Analysis 101: The basic tests
• t-test • paired t-test• Wilcoxon Rank Sum test (Mann-Whitney U test)• Wilcoxon Signed Rank Sum test• Kolmogorov-Smirnov (one and two sample test)• Chi-square test • Fisher’s Exact test• ANOVA• Kruskal-Wallis rank test• repeated measures ANOVA
Why do we need statistics???
When we conduct any type of research, we can make at least two major types of errors when we draw our conclusions:
I)
II)
Why do we need statistics???
When we conduct any type of research, we can make at least two major types of errors when we draw our conclusions:
I) we claim to have found an important treatment effect when in reality there is no treatment effect.
II) we claim that no treatment effect exists when in reality there is an important treatment effect.
Why do we need statistics???
Some important definitions:What is a p-value?
What is power?
Why do we need statistics???
Some important definitions:What is a p-value?
What is power?
P-value: The probability that the difference we observed could be due to chance alone.
Why do we need statistics???
Some important definitions:What is a p-value?
What is power?
P-value: The probability that the difference we observed could be due to chance alone.
Power: The probability that if there is a real difference, our experiment will find it.
Why do we need statistics???
When we conduct any type of research, we can make at least two major types of errors when we draw our conclusions:
I) we can claim to have found an important treatment effect when in reality there is no treatment effect.
II) we can claim that no treatment effect exists when in reality there is an important treatment effect
P-value: The probability that the difference we observed could be due to chance alone.
Power: The probability that if there is a real difference, our experiment will find it.
Sample size calculations: The use of Power
Every experiment should start with a sample size calculation.
• Having adequate power protects us from Type II errors.
Sample size calculations: The use of Power
Every experiment should start with a sample size calculation.
• Having adequate power protects us from Type II errors.
• Forces us to consider a primary outcome for our experiment.
• primary outcomes can be continuous, categorical (interval, ordered, unordered), dichotomous
Sample size calculations: The use of Power
Every experiment should start with a sample size calculation.
• Having adequate power protects us from Type II errors.
• Forces us to consider a primary outcome for our experiment.
• primary outcomes can be continuous, categorical (interval, ordered, unordered), dichotomous
• Should consider issues of design in order to simplify analysis.
Analysis 101: The use of P???
Selection of appropriate study design / analytic technique:
• protects from Type I errors.
• is driven by driven by a combination of study outcome and study design.
Analysis 101: Basics of experimental design
1) Before and after trial• physiological parameter/outcome measured• intervention delivered• physiological parameter/outcome measured again• compare measurement before with measurement after, usually in same subject
Analysis 101: Basics of experimental design
1) Before and after trial• physiological parameter/outcome measured• intervention delivered• physiological parameter/outcome measured again• compare measurement before with measurement after, usually in same subject
2) Comparison between two groups• subjects are randomly assigned to one of two groups• one group receives intervention• compare outcome between two groups after intervention
Analysis 101: Basics of experimental design
1) Before and after trial• physiological parameter/outcome measured• intervention delivered• physiological parameter/outcome measured again• compare measurement before with measurement after, usually in same subject
2) Comparison between two groups• subjects are randomly assigned to one of two groups• one group receives intervention• compare outcome between two groups after intervention
3) Comparison between more than two groups• as above but subjects are assigned to more than two groups• could compare 3 different drugs or 3 different doses
Analysis 101: Outcome identification
Primary outcomes can be 1) continuous, 2) categorical (interval, ordered, unordered), 3) dichotomous
1) Continuous outcomes:• most physiological parameters (Hb, pressures, biochemistry)• usually involves a direct measurement• often Normally distributed
Analysis 101: Outcome identification2) Categorical outcomes:a) interval
• equal unit change between each ordered category
Analysis 101: Outcome identification2) Categorical outcomes:a) interval
• equal unit change between each ordered category• length of stay, age, time to event, some scoring systems• may be Normally distributed
Analysis 101: Outcome identification2) Categorical outcomes:a) interval
• equal unit change between each ordered category• length of stay, age, time to event, some scoring systems• may be Normally distributed
b) ordered• unequal unit change between each ordered category
Analysis 101: Outcome identification2) Categorical outcomes:a) interval
• equal unit change between each ordered category• length of stay, age, time to event, some scoring systems• may be Normally distributed
b) ordered• unequal unit change between each ordered category• most scoring systems, tumor stage or grade, low-medium-high• not usually Normally distributed
Analysis 101: Outcome identification2) Categorical outcomes:a) interval
• equal unit change between each ordered category• length of stay, age, time to event, some scoring systems• may be Normally distributed
b) ordered• unequal unit change between each ordered category• most scoring systems, tumor stage or grade, low-medium-high• not usually Normally distributed
c) unordered• no sequential order to categories
Analysis 101: Outcome identification2) Categorical outcomes:a) interval
• equal unit change between each ordered category• length of stay, age, time to event, some scoring systems• may be Normally distributed
b) ordered• unequal unit change between each ordered category• most scoring systems, tumor stage or grade, low-medium-high• not usually Normally distributed
c) unordered• no sequential order to categories• type of tumor, location, diagnosis• re-think outcome selection!!!!
Analysis 101: Outcome identification3) Dichotomous outcomes:
• only two possible outcome states• tumor / no tumor• dead / alive• follows Binomial distribution
Analysis 101: The basic tests
• t-test • paired t-test• Wilcoxon Rank Sum test (Mann-Whitney U test)• Wilcoxon Signed Rank Sum test• Kolmogorov-Smirnov (one and two sample test)• Chi-square test • Fisher’s Exact test• ANOVA• Kruskal-Wallis rank test• repeated measures ANOVA
Analysis 101: Design and Analysis
1) Before and after trial (same subjects, continuous and interval )
Analysis 101: Design and Analysis
Step 1: Determine if outcome is Normally distributed• plot histogram with density function line
1) Before and after trial (same subjects, continuous and interval )
Analysis 101: Design and Analysis
Step 1: Determine if outcome is Normally distributed• plot histogram with density function line• could ‘formally’ test using Wilkes-Shapiro statistic
120 130 140 150 160 170
hina
0.00
0.02
0.04
0.06
1) Before and after trial (same subjects, continuous and interval )
Analysis 101: Design and Analysis
Step 1: Determine if outcome is Normally distributed• plot histogram with density function line• could ‘formally’ test using Wilkes-Shapiro statistic
120 130 140 150 160 170
hina
0.00
0.02
0.04
0.06
0 200 400 600 800
hicreat
0.000
0.001
0.002
0.003
0.004
0.005
1) Before and after trial (same subjects, continuous and interval )
Analysis 101: Design and Analysis
120 130 140 150 160 170
hina
0.00
0.02
0.04
0.06
0 200 400 600 800
hicreat
0.000
0.001
0.002
0.003
0.004
0.005
1) Before and after trial (same subjects, continuous and interval )
Analysis 101: Design and Analysis
paired t-test Wilcoxon Signed Rank Sum Test
120 130 140 150 160 170
hina
0.00
0.02
0.04
0.06
0 200 400 600 800
hicreat
0.000
0.001
0.002
0.003
0.004
0.005
1) Before and after trial (same subjects, continuous and interval )
Analysis 101: Design and Analysis
paired t-test Wilcoxon Signed Rank Sum Test
NB - if ordered categorical outcome, use one sample Kolmogorov-Smirnov test
120 130 140 150 160 170
hina
0.00
0.02
0.04
0.06
0 200 400 600 800
hicreat
0.000
0.001
0.002
0.003
0.004
0.005
1) Before and after trial (same subjects, continuous and interval )
Analysis 101: Design and Analysis
2) Comparison between two groups(continuous and interval)
Analysis 101: Design and Analysis
2) Comparison between two groupsStep 1: Determine if outcome is Normally distributed
• plot histogram (use all data) with density function line• could ‘formally’ test using Wilkes-Shapiro statistic
0 10 20 30 40 50
apache2
0.00
0.01
0.02
0.03
0.04
(continuous and interval)
Analysis 101: Design and Analysis
2) Comparison between two groupsStep 1: Determine if outcome is Normally distributed
• plot histogram (use all data) with density function line• could ‘formally’ test using Wilkes-Shapiro statistic
0 200 400 600 800
hicreat
0.000
0.001
0.002
0.003
0.004
0.005
0 10 20 30 40 50
apache2
0.00
0.01
0.02
0.03
0.04
(continuous and interval)
Analysis 101: Design and Analysis
2) Comparison between two groups
0 200 400 600 800
hicreat
0.000
0.001
0.002
0.003
0.004
0.005
0 10 20 30 40 50
apache2
0.00
0.01
0.02
0.03
0.04
(continuous and interval)
Analysis 101: Design and Analysis
2) Comparison between two groups
0 200 400 600 800
hicreat
0.000
0.001
0.002
0.003
0.004
0.005
t-test Wilcoxon Rank Sum test
0 10 20 30 40 50
apache2
0.00
0.01
0.02
0.03
0.04
(continuous and interval)
Analysis 101: Design and Analysis
2) Comparison between two groups
0 200 400 600 800
hicreat
0.000
0.001
0.002
0.003
0.004
0.005
t-test Wilcoxon Rank Sum test
NB - if ordered categorical outcome, use two sample Kolmogorov-Smirnov test
0 10 20 30 40 50
apache2
0.00
0.01
0.02
0.03
0.04
(continuous and interval)
Analysis 101: Design and Analysis
3) Comparison between more than two groups
Analysis 101: Design and Analysis
3) Comparison between more than two groupsStep 1: Determine if outcome is Normally distributed
• plot histogram (use all data) with density function line• could ‘formally’ test using Wilkes-Shapiro statistic
120 130 140 150 160 170
hina
0.00
0.02
0.04
0.06
0 200 400 600 800
hicreat
0.000
0.001
0.002
0.003
0.004
0.005
Analysis 101: Design and Analysis
3) Comparison between more than two groups
120 130 140 150 160 170
hina
0.00
0.02
0.04
0.06
0 200 400 600 800
hicreat
0.000
0.001
0.002
0.003
0.004
0.005
Analysis 101: Design and Analysis
3) Comparison between more than two groups
ANOVA Kruskal-Wallis rank test
120 130 140 150 160 170
hina
0.00
0.02
0.04
0.06
0 200 400 600 800
hicreat
0.000
0.001
0.002
0.003
0.004
0.005
Analysis 101: Design and Analysis
3) Comparison between more than two groups
ANOVA Kruskal-Wallis rank testNB - could transform (calculate the log or ln) each outcome value and redo histogram…. if transformed values are Normally distributed, can now use ‘more powerful’ ANOVA (or t-test if 2 samples).
120 130 140 150 160 170
hina
0.00
0.02
0.04
0.06
0 200 400 600 800
hicreat
0.000
0.001
0.002
0.003
0.004
0.005
Analysis 101: Dichotomous outcomes1) Before and after trial
• rate before intervention compared to rate after intervention• McNemer’s chi-square
Analysis 101: Dichotomous outcomes1) Before and after trial
• rate before intervention compared to rate after intervention• McNemer’s chi-square
2) Comparison between two groups• create 2x2 table, calculate rate for each Group
Dead AliveGroup A 2 8 20% mortalityGroup B 7 3 70% mortality• compare using chi-square test
Analysis 101: Dichotomous outcomes1) Before and after trial
• rate before intervention compared to rate after intervention• McNemer’s chi-square
2) Comparison between two groups• create 2x2 table, calculate rate for each Group
Dead AliveGroup A 2 8 20% mortalityGroup B 7 3 70% mortality• compare using chi-square testNB - if any one cell contains < 5 counts, use Fisher’s Exact test
Analysis 101: Dichotomous outcomes1) Before and after trial
• rate before intervention compared to rate after intervention• McNemer’s chi-square
2) Comparison between two groups• create 2x2 table, calculate rate for each Group
Dead AliveGroup A 2 8 20% mortalityGroup B 7 3 70% mortality• compare using chi-square testNB - if any one cell contains < 5 counts, use Fisher’s Exact test
3) Comparison between more than two groups• undertake a series of comparisons via 2x2 tables as above
Analysis 101: Special considerationsTransformations:
Sometimes its possible to ‘transform’ a long tailed distribution to a normal distribution.Calculate the log or ln of each outcome value and redo histogram.Allows us to apply ‘more powerful’ tests based on assumption of Normality (paired t-test, t-test, ANOVA).Try non-parametric test first <- fewer assumptions!!!!
0 200 400 600 800
hicreat
0.000
0.001
0.002
0.003
0.004
0.005
Analysis 101: Special considerations
The t-test has 3 basic, fundamental underlying assumptions:
1) Outcomes are Normally distributed
• test assumptions of Normality
• use non-parametric tests
Analysis 101: Special considerations
The t-test has 3 basic, fundamental underlying assumptions:
1) Outcomes are Normally distributed
• test assumptions of Normality
• use non-parametric tests
2) Outcomes are independent
• if outcomes are from same subjects, use paired t-test
Analysis 101: Special considerations
The t-test has 3 basic, fundamental underlying assumptions:
1) Outcomes are Normally distributed
• test assumptions of Normality
• use non-parametric tests
2) Outcomes are independent
• if outcomes are from same subjects, use paired t-test
3) The variance of each group is similar
• stats package should formally test equality of variances
• different p-values for each condition
Analysis 101: Summary
• t-test (two groups, Normally distributed)• paired t-test (before/after, Normally distributed)• Wilcoxon Rank Sum test (two groups, non-parametric)• Wilcoxon Signed Rank Sum test (before/after, non-parametric)• Kolmogorov-Smirnov (before/after, two groups, ordered categorical)• Chi-square test (dichotomous outcome)• Fisher’s Exact test (dichotomous outcome, any cell size < 5)• ANOVA (more than two groups, Normally distributed)• Kruskal-Wallis rank test (more than two groups, non-parametric)• repeated measures ANOVA
www.EvidenceBased.net/talks