Significance, P-value and t-tests

Question: Whether an observation is significantly different?

Solution: Hypothesis testing

1. State the null hypothesis(H0)

2. Choose appropriate test ,test statistic T

3. Derive the distribution of T under H0

4. Compute the observed value tobs of T

5. Calculate the p-value

6. Reject H0 if the p-value is less than the selected significance

P-value tells us how strong the evidence is against H0.

P-value is not the probability that H0 is True

To calculate P-value, we need to know the distribution under H0

• Normal distribution

The t-distribution

t-distribution take into account that most sample will underestimate population’s variability

Interpreting P Values

Compatibility

“A P value measures a sample’s compatibility with a hypothesis, not the truth of the hypothesis”

Requires combinations of other measures to make correct decisions

Bayesian cutoffs

P-value-based FDR calculations (eFDR)

P values and effect sizes

Bayesian Analysis

Bayes factor: B Ratio of average likelihood under alternative hypothesis and null

hypothesis

Requires specification of a prior distribution

Upper bound for Bayes factor: 𝐵 Provides the maximum alternative to null likelihood ratio

𝐵 ≥ 20 indicates strong evidence for alternative hypothesis

𝐵 ≤ −1/(𝑒𝑃𝑙𝑛(𝑃)) 𝑃 < 𝛼 = 0.05 → 𝐵 ≤ 2.5

𝑃 < 𝛼 = 0.025 → 𝐵 ≤ 3.9

𝐵 > 20 → 𝛼 < 0.0032

False Discovery Rate

FDR is the expected proportion of rejected null hypotheses that are false rejections

Let 𝜋0 represent the true proportion of tests that are truly null 𝛼𝜋0 is the expected number of false rejections

𝛽(1 − 𝜋0) is the number of non-null tests that we end up rejecting, where 𝛽 is the power of the test (𝛽 = 𝑃(𝐻0 𝑟𝑒𝑗𝑒𝑐𝑡𝑒𝑑|𝐻1 𝑡𝑟𝑢𝑒))

i.e. the number of correct rejections

𝑒𝐹𝐷𝑅 = 𝛼𝜋0/(𝛼𝜋0+ 𝛽(1 − 𝜋0) ) Reasonable estimate for FDR

Effect Size Strong support for consideration of effect size when interpreting

P values

Provide confidence interval for parameter of interest Based on confidence level

P Values

Random Variables Random samples result in random distribution for P values

Null true

P value ~ Unif(0,1), 𝜇 = 0.5, 𝜎 = 0.29

Alternative true

Increased power results in decreased variability