Reading Testing a point-null hypothesis, by Jiahuan Li, Feb. 25, 2013

Testing a Point Null Hypothesisi: TheIrreconcilability of P Values and Evidence

JAMES O.BERGER and THOMAS SELLKE

25.02.2013

JAMES O.BERGER and THOMAS SELLKE Testing a Point Null Hypothesisi: The Irreconcilability of P Values and Evidence

Content

1 INTRODUCTION

2 POSTERIOR PROBABILITIES AND ODDS3 LOWER BOUNDS ON POSTERIOR PROBABILI-

TIESIntroductionLower Bounds for GA ={All Distributions}Lower Bounds for GS ={Symmetric Distributions}Lower Bounds for GUS ={Unimodal,Symmetric Distributions}Lower Bounds for GNOR ={Normal Distributions}

4 MORE GENERAL HYPOTHESES AND CONDITIONALCALCULATOINS

General FormulationMore General Hypotheses

5 CONCLUSIONS


Content

1 INTRODUCTION2 POSTERIOR PROBABILITIES AND ODDS

3 LOWER BOUNDS ON POSTERIOR PROBABILI-TIES

IntroductionLower Bounds for GA ={All Distributions}Lower Bounds for GS ={Symmetric Distributions}Lower Bounds for GUS ={Unimodal,Symmetric Distributions}Lower Bounds for GNOR ={Normal Distributions}



5 CONCLUSIONS


Content

1 INTRODUCTION2 POSTERIOR PROBABILITIES AND ODDS3 LOWER BOUNDS ON POSTERIOR PROBABILI-

TIESIntroductionLower Bounds for GA ={All Distributions}Lower Bounds for GS ={Symmetric Distributions}Lower Bounds for GUS ={Unimodal,Symmetric Distributions}Lower Bounds for GNOR ={Normal Distributions}



5 CONCLUSIONS


1. Introduction

The paper studies the problem of testing a point null hypothesis,of interest is the relationship between the P value and conditionaland Bayesian measures of evidence against the null hypothesis

∗ The overall conclusion is that P value can be highlymisleading measures of the evidence provided by le dataagainst the null hypothesis


1. Introduction

Consider the simple situation of observing a random quantityX having density f (x | θ) , θ ⊂ R1 , it is desired to test thenull hypothesis H0 : θ = θ0 versus the alternative hypothesisH1 : θ 6= θ0.

p = Prθ=θ0(T (X ) ≥ T (x))


1. lntroduction

Example

Suppose that X = (X1, .......,Xn) where the Xi are iid N(θ, σ2)Then the usual test statistic is

T (X ) =√n | X̄ − θ0 | /σ

where X̄ is the sample mean, and

p = 2(1− Φ(t))

where Φ is the standard normal cdf and

t = T (x) =√n | x̄ − θ0 | /σ


1. Introduction

We presume that the classical approcach is the report of p,rather than the report of a Neyman-Perason error probability.This is because

Most statistician prefer use of P values, feeling it to be impor-tant to indicate how strong the evidence against H0 is .

The alternative measures of evidence we consider are based onknowledge of x.

There are several well-known criticisms of testing a point nullhypothesis.

One is the issue of ’statistical’ versus ’practical’ significance,that one can get a very small p even when | θ− θ0 | is so smallas to make θ equivalent to θ0 for practical purposes.

Another well known is ’Jeffrey’s paradox’


1. Introduction

Example

Consider a Bayesian who chooses the prior distribution on θ, whichgives probability 0,5 to H0 and H1 and spreads mass out on H1

according to an N(θ, σ2) density. It will be seen in Section 2 thatthe posterior probability, Pr(H0 | x), of H0 is given by

Pr(H0 | x) = (1 + (1 + n)−1/2exp{t2/[2[(1 + 1/n)]})−1


1. Introduction

Table 1 : Pr(H0 | x) for Jeffreys-Type Prior)

n

p t 1 5 10 20 50 100 1,000

.10 1.645 .42 .44 .47 .56 .65 .72 .89

.05 .1.960 .35 .33 .37 .42 .52 .60 .82

.01 .2.576 .21 .13 .14 .16 .22 .27 .53.001 3.291 .086 .026 .024 .026 .034 .045 .124

The conflict between p and Pr(H0 | x) is apparent.


1. Introduction

Example

Again consider a Bayesian who gives each hypothesis prior probabil-ity 0.5, but now suppose that he decides to spread out the mass onH1 in the symmetric fashion that is as favorable to H1 as possible.The corresponding values of Pr(H0 | x) are determined in Section 3and are given in Table 2 for certain values of t.

Table 2 : Pr(H0 | x) for a Prior Biased Towar H1

P value(p) t Pr(H0 | x)

.10 1.645 .340

.05 .1.960 .227

.01 .2.576 .068.001 3.291 .0088


1. Introduction

Example (A Likelihood Analysis)

It is common to perceive the comparative evidence provided by x fortwo pssible parameter values, θ1andθ2, as being measured by thelikelihood ratio

lx(θ1 : θ2) = f (x | θ1)/f (x | θ2)

A lower bound on the comparative evidence would be

lx = infθlx(θ0 : θ) =

f (x | θ0)

supθ f (x | θ)= exp{−t2/2}


1. Introduction

Values of lx for various t are given in Table 3

Table 3 : Bounds on the Comparative Likelihood

Likelihood ratioP value(p) t lower bound (lx)

.10 1.645 .340

.05 .1.960 .227

.01 .2.576 .068.001 3.291 .0088


2. Posterior probabilities and odds

let 0< π0 < 1 denote the prior probability of H0, and let π1 = 1−π0

denote the prior probability of H1, suppose that the mass on H1 isspread out according to the density g(θ).

Realistic hypothesis: H0 :| θ − θ0 |≤ b

Prior probability π0 would be assigned to {θ :| θ − θ0 |≤ b}

(To a Bayesian, a point null test is typically reasonable only whenthe prior distribution is of this form)



The posterior odds ratio of H0 to H1

Pr(H0 | x)

1− Pr(H0 | x)︸︷︷︸Post odds

=π0

(1− π0)︸︷︷︸Prior odds

× f (x | θ0)

mg (x)︸︷︷︸Bayes factor Bg (x)

Interest in the Bayes factor centers around the fact that it doesnot involve the prior probabilities of the hypotheses and hence issometimes interpreted as the actual odds of the hypotheses impliedby the data alone.



Example (Jeffreys-Lindley paradox)

Suppose that π0is arbitrary and g is again N(θ0, σ2).Since a suffi-

cent statistic for θ is X̄ v N(θ0, σ2/n) ,we have that mg (x̄) is an

N(θ0, σ2(1 + n−1)) distribution. Thus

Bg (x) = f (x | θ0)/mg (x̄)

=[2πσ2/n]−2exp{−n

2 (x̄ − θ0)2/σ2}[2πσ2(1 + n−1)]−1/2exp{−1

2 (x̄ − θ20)/[σ2(1 + n−1]}

= (1 + n)1/2exp{−1

2t2/(1 + n−1)}

andPr(H0 | x) = [1 + (1− π0)/(π0Bg )]−1

= [1 +(1− π0)

π0(1 + n)−1/2 × exp{1

2t2/(1 + n−1)}]−1


3. Lower bounds on posterior probabilites3.1 Introduction

This section will examine some lower bounds on Pr(H0 | x)when g(θ), the distribution of θ given that H1 is true isallowed to vary within some class of distribitions G

GA ={all distributions}GS ={all distributions symmetric about θ0}GUS={all unimodal distribution symmetric about θ0}GNOR={all N(θ0, τ

2)distributions, 0≤ τ2 <∞}

Even though these G’s are supposed to consist only of distributionon {θ | θ 6= θ0 }, il will be convenient to allow them to includedistributions with mass at θ0, so the lower bounds we compute arealways attained.


3. Lower bounds on posterior probabilites3.1 Introduction

LettingPr(H0 | x ,G ) = inf

g∈GPr(H0 | x)

andB(xmG ) = inf

g∈GBg (x)

we see immediately form formulas before that

B(x ,G ) = f (x | θ0)/ supg∈G

mg (x)

and

Pr(H0 | xmG ) = [1 +(1− π0)

π0× 1

B(x ,G )]−1


3. Lower bounds on posterior probabilites3.2 Lower bounds for GA={All distributions}

Theorem

Suppose that a maximum likelihood estimate of θ0, exists for theobserved x. Then

B(x ,GA) = f (x | θ0)/f (x | θ̂(x))

and

Pr(H0 | x ,GA) = [1 +(1− π0)

π0× f (x | θ̂(x))

f (x | θ0)]−1



Example

In this situation,we have

B̄(x ,GA) = e−t2/2

and

Pr(H0 | x ,GA) = [1 +(1− π0)

π0et

2/2]−1

For servral choices of t, Table 4 gives the corresponding two-sidedP values,p, and the values of Pr(H0 | x ,GA),with π0 = 0.5.



For servral choices of t, Table 4 gives the correspondingtwo-sided P values,p, and the values of Pr(H0 | x ,GA),withπ0 = 0.5.

Table 4 : Comparison of P values and Pr(H0 | x ,GA) when π0 = 0.5

P value(p) t Pr(H0 | x ,GA) Pr(H0 | x ,GA)/(pt)

.10 1.645 .205 1.25

.05 .1.960 .128 1.30

.01 .2.576 .035 1.36.001 3.291 .0044 1.35



Theorem

For t > 1.68 and π0 = 0.5 in Example 1,

Pr(H0 | x ,GA)/pt >√π/2 u 1.253

Furthermorelimt→∞

Pr(H0 | x ,GA)/pt =√π/2


3. Lower bounds on posterior probabilites3.3 Lower bounds for GS={Symmetric distributions}

There is a large gap between Pr(H0 | x ,GA)andPr(H0 | x) forthe Jeffreys-type single prior analysis.This reinforces thesuspicion that using GA unduly biases the conclusion againstH0 and suggests use of more reasonable classes of priors.

Theorem

supg∈G2ps

mg (x) = supg∈GS

mg (x),

soB(x ,G2PS) = B(x ,GS)

andPr(H0 | x ,G2ps) = Pr(H0 | x ,GS)



Example

If t ≤ 1, a calculus argument show that the symmetric two pointdistribution that strictly maximizes mg (x) is the degenerate ”two-point”distribution putting all mass at θ0. Thus B(x ,GS) = 1 andPr(H0 | x ,GS) = π0 for t ≤ 1.If t ≥ 1 , then mg (x) is maximized by a nondegenerate elementof G2ps . For moderately large t, the maximum value of mg (x) forg∈ G2ps is very well approximated by taking g to be the two-pointdistribution putting equal mass at θ̂(x) and at 2θ − θ̂(x). so

B(x ,GS) uϕ(t)

0.5ϕ(0) + 0.5ϕ(2t)u 2 exp {−0.5t2}



Example

For t ≤ 1.645, the first approximation is accurate to within 1 in thefourth significant digit and the second approximation to within 2 inthe third significant digit.Table 5 gives the value of Pr(H0 | x ,Gs) of several choices of t.

Table 5 : Comparison of P values and Pr(H0 | x ,GS) when π0 = 0.5

P value(p) t Pr(H0 | x ,GS) Pr(H0 | x ,GS)/(pt)

.10 1.645 .340 2.07

.05 .1.960 .227 2.31

.01 .2.576 .068 2.62.001 3.291 .0088 2.68


3. Lower bounds on posterior probabilites3.4 Lower bounds for GUS={Unimodal, Symmetric distributions}

Minimizing Pr(H0 | x) over all symmetric priors still involvesconsiderable bias against H0. A further ’objective’ restriction,which would seem reasonable to many, is to require the priorto be unimodal, or non-increasing in | θ − θ0 | .

Theorem

supg∈Gus

mg (x) = supg∈US

mg (x),

with US={all symmetric uniform distributions}so B(x ,GUS) = B(x ,US) and Pr(H0 | x ,GUS) = Pr(H0 | x ,US)



Theorem

If t ≤ 1 in example 1, then B(x ,GUS) = 1 and Pr(H0 | x .GUS) = π0.Ift > 1 then

B(x .GUS) =2ϕ(t)

ϕ(K + t) + ϕ(K − t)

and

Pr(H0 | x ,GUS) = [1 +(1− π0)

π0× ϕ(K + t) + ϕ(K = t)

2ϕ(t)]−1

where K > 0

Figures 1 and 2 give values of K and B for various val-ues of t in this problem



Table 6 gives Pr(H0 | x ,GUS) for some specific importantvalues of t and π0 = 0.5

Table 6 : Comparison of P values and Pr(H0 | x ,GUS) when π0 = 0.5

P value(p) t Pr(H0 | x ,GUS) Pr(H0 | x ,GUS)/(pt)

.10 1.645 .390 1.44

.05 .1.960 .290 1.51

.01 .2.576 .109 1.64.001 3.291 .018 1.66


3. Lower bounds on posterior probabilites3.5 Lower bounds for GNOR={Normal distributions}

We have seene that minimizing Pr(H | x)overg ∈ GUS is the sameas minimizing over g ∈ US . Althought using US is much morereasonable than using GA,there is still some residual bias againstH0 involved in using US .

Theorem

If t ≤ 1in Example 1, then B(x ,GNOR) = 1 and Pr(H0 | x ,GNOR) =π0. If t > 1, then

B(x ,GNOR) =√ete−t

2/2

and

Pr(H0 | x ,GNOR) = [1 +(1− π0)

π0× exp{t2/2}√

et]−1


3. Lower bounds on posterior probabilites3.5 Lower bounds for GNOR={Normal distributions}

Table 7 gives Pr(H0 | x ,GNOR) for servral values of t

Table 7 : Comparison of P values and Pr(H0 | x ,GNOR) when π0 = 0.5

P value(p) t Pr(H0 | x ,GNOR) Pr(H0 | x ,GNOR)/(pt)

.10 1.645 .412 1.52

.05 .1.960 .321 1.67

.01 .2.576 .133 2.01.001 3.291 .0235 2.18


4. More general hypotheses and conditional calculations4.1 General formulation

Consider the Bayesian calculation of Pr(H0 | A) , where H0 isof the form H0 : θ ∈ Θ0 and A is the set in which x is knownto reside. Then letting π0 and π1 denote the priorprobabilities of H0 and H1 and g1 and g2 as the densities onΘ0 and Θ1. Then we have :

Pr(H0 | A) = [1 +1− π0

π + 0× mg1(A)

mg0(A)]−1

where

mg1(A) =

∫Θ0

Prθ(A)gi (θ)dθ


4. More general hypotheses and conditional calculations4.1 General formulation

For the general formulation, one can determine lower boundson Pr(H0 | A) by choosing sets G0 and G1 of g0 and g1,respectively, calculating

B(A,G0,G1) = infg0∈G0

mg0(A)/ supg1∈G1

mg1(A)

and defining Pr(H0 | A,G0,G1) = [1 + (1−π0)π0× 1

B(A,G0,G1) ]−1


4. More general hypotheses and conditional calculations4.2 More general hypotheses

Assume in this section that A= {x}. The lower bounds canbe applied to a variety of generalizations of point nullhypotheses. If Θ0 is a small set about θ0, the negeral lowerbounds turn out to be essentially equivalent to the point nulllower bounds.

In example 1, suppose that the hypotheses were H0 : θ ∈ (θ0 −b, θ0 + b) and H1 : θ /∈ (θ0 − b, θ0 + b) If | t −

√nb/σ |≥ 1

and G0 = G1 = GS , then B(x ,G0,G1) and Pr(H0 | x ,G0,G1) areexactly the same as B and P for testing the point null.


5. Conclusion

B(x ,GUS) and the P value calculated at (t − 1)+ in instead of t,this last will be called the P value of (t − 1)+ Figure shows thatthis comparative likelihood is close to the P value that would beobtained if we replaced t by (t − 1)+. The implication is that :t = 1 means only mild evidence against H0, t = 2 meanssignificant evidence against H0, and t = 3 means highly evidenceagainst H0, and t = 4 means over whelming evidence against H0,should at least replaced by the rule of thumb, that is :t = 1 means no evidence against H0, t = 2 means only mildevidence against H0, and t = 3 means significant evidence againstH0, and t = 4 means highly significant evidence against H0.


5. Conclusion

t = 1 means no evidence against H0, t = 2 means only mildevidence against H0, and t = 3 means significant evidence against

H0, and t = 4 means highly significant evidence against H0.


References

EDWARDS, W, LINDMAN, H, and SAVAGE, L.J (1963).Bayesian statistical inference for psychological research. Psycho-logical Review 70, 193-242Jayanta Ghosha, Sumitra Purkayastha and Tapas SamantaRole of P-values and other Measures of Evidence in Bayesian Anal-ysis. Handbook of Statistics, Vol. 25James O. Berger Statistical Decision Theory and Bayesian Analy-sis. Springer Series in Statistics


Reading Testing a point-null hypothesis, by Jiahuan Li, Feb. 25, 2013

Education

Transcript of Reading Testing a point-null hypothesis, by Jiahuan Li, Feb. 25, 2013