S ta tis tic s o f S m ll S -...
Transcript of S ta tis tic s o f S m ll S -...
Gary F
eldm
an N
EP
PS
R 23 A
ug
ust 2002 1
Statistics o
f Sm
all Sig
nals
lA
partial d
iscussio
n o
f a pap
er “Un
ified A
pp
roach
to th
e Classical S
tatistical An
alysis of S
mall
Sig
nals,” w
hich
I wro
te with
Bo
b C
ou
sins. [P
hys.
Rev. D
57, 3873 (1988)]
Gary F
eldm
an N
EP
PS
R 23 A
ug
ust 2002 2
A S
imp
le Exam
ple (1)
lS
up
po
se you
are searchin
g fo
r a rare pro
cess and
have a
well-kn
ow
n exp
ected b
ackgro
un
d o
f 3 events, an
d yo
uo
bserve 0 even
ts. Wh
at 90% co
nfid
ence lim
it can yo
u set o
nth
e un
kno
wn
rate for th
is rare pro
cess?l
A classical (o
r frequ
entist) statistician
makes a statem
ent
abo
ut th
e pro
bab
ility of d
ata given
theo
ry. T
hat is, g
iven a
hyp
oth
esis for th
e value o
f an u
nkn
ow
n tru
e value µ, h
e or
she w
ill give yo
u th
e pro
bab
ility of o
btain
ing
a set of d
ata x,P
(x | µ).
lA
classical con
fiden
ce interval (Jerzy N
eyman
, 1937) is a
statemen
t of th
e form
: Th
e un
kno
wn
true valu
e of µ lies in
the reg
ion [µ
1 ,µ2 ]. If th
is statemen
t is mad
e at the 90%
con
fiden
ce level, then
it will b
e true 90%
of th
e time, an
dfalse 10%
of th
e time.
Gary F
eldm
an N
EP
PS
R 23 A
ug
ust 2002 3
A S
imp
le Exam
ple (2)
lP
oisso
n statistics P
(x = 0 | µ
= 2.3) =
0.1. Th
erefore, in
the
“stand
ard” classical ap
pro
ach, µ
< 2.3 at 90%
C.L
. Sin
ce µ =
s + b
, and
b =
3.0, s <
-0.7 at 90% C
.L.
lT
hu
s, we are led
to a statem
ent th
at we kn
ow
is a prio
ri false.
Gary F
eldm
an N
EP
PS
R 23 A
ug
ust 2002 4
Bayesian
Statistics
lA
Bayesian
takes the o
pp
osite
po
sition
from
a classicalstatistician
. He o
r she calcu
lates the p
rob
ability o
f theo
ryg
iven d
ata. Th
at is, given
a set of d
ata x, he o
r she w
illcalcu
late the p
rob
ability th
at the u
nkn
ow
n tru
e value is µ
,P
(µ | x).
lT
his ap
pears attractive b
ecause it is w
hat yo
u really w
ant to
kno
w. H
ow
ever, it com
es at a price:
Gary F
eldm
an N
EP
PS
R 23 A
ug
ust 2002 5
Bayes’s T
heo
rem
lP
(x | µ) an
d P
(µ | x
) are related
by B
ayes’s T
heo
rem, w
hich
inset th
eory is th
e statemen
t that an
elemen
t is in b
oth
A an
d B
isP
(A | B
) P(B
) = P
(B | A
) P(A
)w
hich
for p
rob
abilities b
ecom
esP
(µ | x) =
P(x | µ
) P(µ
)/P(x).
P(x) is ju
st a no
rmalizatio
n term
, bu
t Bayes
’s Th
eorem
transfo
rms P
(µ), th
e prio
r distrib
utio
n o
f “deg
ree of b
elief” inµ
, to P
(µ | x), th
e po
sterior d
istribu
tion
.
A “cred
ible in
terval” or “B
ayesian co
nfid
ence in
terval” isfo
rmed
by
P
(m|x
)dm
m1
m2
Ú=
90
%.
Gary F
eldm
an N
EP
PS
R 23 A
ug
ust 2002 6
An
Exam
ple (1)
lS
up
po
se you
have a larg
e nu
mb
er of m
arbles, w
hich
areeith
er wh
ite or b
lack, and
you
wish
info
rmatio
n o
n th
efractio
n th
at are wh
ite, µ. Y
ou
draw
a sing
le marb
le, and
it isw
hite. W
hat can
you
say at 90% co
nfid
ence?
Classical:
µ ≥ 0.1
Prio
rB
ayesian:
flatµ
≥ 0.316µ
µ ≥ 0.464
1/µµ
≥ 0.1(1-µ)
µ ≥ 0.196
1/(1-µ)
un
no
rmalizab
le
Gary F
eldm
an N
EP
PS
R 23 A
ug
ust 2002 7
An
Exam
ple (2)
lN
otice th
at mo
st of th
e Bayesian
prio
rs do
no
t cover, i.e.,
they are n
ot tru
e statemen
ts the stated
fraction
of th
e time
(90% in
this case). T
here is n
o req
uirem
ent th
at credib
lein
tervals cover. H
ow
ever, Bo
b C
ou
sins w
arns [A
m. J. P
hys.
63, 398 (1995)],
“…if a B
ayesian m
etho
d is kn
ow
n to
yield in
tervals with
frequ
entist co
verage ap
preciab
ly less than
the stated
C.L
. for
som
e po
ssible valu
e of th
e un
kno
wn
param
eters, then
itseem
s to h
ave no
chan
ce of g
ainin
g co
nsen
sus accep
tance
in p
article ph
ysics.”
Gary F
eldm
an N
EP
PS
R 23 A
ug
ust 2002 8
Th
e Ro
le of
Bayesian
Statistics (1)
lH
arrison
Pro
sper [P
hys. R
ev. D 37, 1153 (1988)] arg
ues fo
r a1/µ
prio
r based
on
a scaling
argu
men
t. I fou
nd
itu
nsatisfacto
ry for tw
o reaso
ns:
lIt fails fo
r x =
0. (Un
no
rmalizab
le)l
In g
eneral, it u
nd
ercovers.
lT
o q
uo
te Bo
b’s p
rose fro
m o
ur p
aper:
l“In
ou
r view, th
e attemp
t to fin
d a n
on
-info
rmative p
rior
with
in B
ayesian in
ference is m
isgu
ided
. Th
e real po
wer
of B
ayesian in
ference lies in
its ability to
inco
rpo
rate‘in
form
ative’ p
rior in
form
ation
, no
t ‘ign
oran
ce.’”
Gary F
eldm
an N
EP
PS
R 23 A
ug
ust 2002 9
Th
e Ro
le of
Bayesian
Statistics (2)
lP
rosp
er wro
te that h
e was u
sing
a Bayesian
app
roach
becau
se
l“…
we are m
erely ackno
wled
gin
g th
e fact that a co
heren
tso
lutio
n to
the sm
all-sign
al pro
blem
is mo
re easilyach
ieved w
ithin
a Bayesian
framew
ork th
an o
ne w
hich
uses th
e meth
od
s of ‘classical’ statistics.”
lT
hro
ug
h th
is talk, I ho
pe to
con
vince yo
u th
at this is n
olo
ng
er true.
Gary F
eldm
an N
EP
PS
R 23 A
ug
ust 2002 10
Co
nstru
ction
of
Co
nfid
ence In
tervals
lN
eyman
’s prescrip
tion
: Befo
re do
ing
an exp
erimen
t, for
each p
ossib
le value o
f theo
ry param
eters determ
ine a reg
ion
of d
ata that o
ccurs C
.L. o
f the tim
e, say 90%
. After d
oin
g th
eexp
erimen
t, find
all of
values o
f the th
eory p
ar-am
eters for w
hich
you
rd
ata is in th
eir 90%reg
ion
. Th
is is the
con
fiden
ce interval.
lN
otice th
at there is co
m-
plete freed
om
of ch
oice
of w
hich
90% to
cho
ose.
Th
is will b
e the key to
ou
rso
lutio
n.
Gary F
eldm
an N
EP
PS
R 23 A
ug
ust 2002 11
Exam
ples o
f Po
isson
Co
nfid
ence B
elts
• Fo
r ou
r examp
le: 90% C
.L. lim
its for P
oisso
n µ
with
backg
rou
nd
= 3
Up
per lim
itsC
entral lim
its
Gary F
eldm
an N
EP
PS
R 23 A
ug
ust 2002 12
Th
e So
lutio
n
lF
or b
oth
the u
pp
er limit an
d cen
tral limit, x =
0 exclud
es the
wh
ole p
lane. B
ut co
nsid
er the p
rob
lem fro
m th
e po
int o
fview
of th
e data. If o
ne m
easures n
o even
ts, then
clearly the
mo
st likely value o
f µ is zero
. Wh
y sho
uld
on
e rule o
ut th
em
ost likely scen
ario?
lT
herefo
re, we p
rop
osed
a new
ord
ering
prin
ciple b
ased o
nth
e ratio o
f a given
µ to
the m
ost likely µ
:
wh
ere µ* is th
e mo
st likely value o
f µ g
iven x.
†
R=
P(x
|m)
P(x
|m*)
Gary F
eldm
an N
EP
PS
R 23 A
ug
ust 2002 13
An
Exam
ple (1)
lE
xamp
le for µ
= 0.5 an
d b
= 3:
x0.121
0.1405.0
0.0178
x x
0.2590.149
4.00.039
7
x x
70.480
0.1613.0
0.0776
x x
40.753
0.1752.0
0.1325
x x
10.966
0.1951.0
0.1894
x x
20.963
0.2240.0
0.2163
x x
30.826
0.2240.0
0.1852
x x
50.708
0.1490.0
0.1061
60.607
0.0500.0
0.0300
C.L
.U
.L.
rank
RP
(x|µ
*)µ
*P
(x|µ
)x
Gary F
eldm
an N
EP
PS
R 23 A
ug
ust 2002 14
Un
ified P
oisso
n L
imits
l90%
C.L. unified
limits for P
oisson µ w
ith background = 3
• So
lutio
n to
ou
r orig
inal p
rob
lem: µ
< 1.08 at 90%
C.L
.
Gary F
eldm
an N
EP
PS
R 23 A
ug
ust 2002 15
Exam
ples o
f Gau
ssianC
on
fiden
ce Belts
• 90% C
.L. limits for G
aussian µ ≥ 0 vs. x (total – background) in s
Up
per L
imits
Cen
tral Lim
its
Gary F
eldm
an N
EP
PS
R 23 A
ug
ust 2002 16
Flip
-Flo
pp
ing
(1)
lH
ow
do
es a typical p
hysicist u
se these p
lots?
l“If th
e result x <
3s, I w
ill qu
ote an
up
per lim
it.”
l“If th
e result x >
3s, I w
ill qu
ote a cen
tral con
fiden
cein
terval.”
l“If th
e result x <
0, I will p
retend
I measu
red zero
.”
Gary F
eldm
an N
EP
PS
R 23 A
ug
ust 2002 17
Flip
-Flo
pp
ing
(2)
lT
his resu
lts in th
e follo
win
g:
lIn
the ran
ge 1.36 ≤ µ
≤ 4.28, there is o
nly 85%
coverag
e!l
Du
e to flip
-flop
pin
g (d
ecidin
g w
heth
er to u
se an u
pp
er limit
or a cen
tral con
fiden
ce regio
n b
ased o
n th
e data) th
ese aren
ot valid
con
fiden
ce intervals.
Gary F
eldm
an N
EP
PS
R 23 A
ug
ust 2002 18
Un
ified S
olu
tion
for
the G
aussian
Case (1)
lN
otes:
lT
his ap
pro
aches th
e central lim
its for x >
>1
lT
he u
pp
er limit fo
r x = 0 is 1.64, th
e two
-sided
rather th
anth
e on
e-sided
limit.
Gary F
eldm
an N
EP
PS
R 23 A
ug
ust 2002 19
Un
ified S
olu
tion
for
the G
aussian
Case (2)
lN
otes (co
ntin
ued
):
lF
rom
the d
efinin
g 1937 p
aper o
f Neym
an, th
is is the o
nly
valid co
nfid
ence b
elt, since th
ere are 4 requ
iremen
ts for a
valid b
elt:
(1) It mu
st cover.
(2) Fo
r every x, there m
ust b
e at least on
e µ.
(3) No
ho
les (on
ly valid fo
r sing
le µ).
(4) Every lim
it mu
st inclu
de its en
d p
oin
ts.
Gary F
eldm
an N
EP
PS
R 23 A
ug
ust 2002 20
Sen
sitivity
lT
he m
ain o
bjectio
n to
this w
ork h
as been
that an
experim
ent
that o
bserves few
er events th
an th
e expected
backg
rou
nd
may rep
ort a lo
wer u
pp
er limit th
an a (b
etter desig
ned
?)
experim
ent th
at has n
o b
ackgro
un
d.
lT
o ad
dress th
is pro
blem
and
to p
rovid
e add
ition
alin
form
ation
for th
e reader’s assessm
ent o
f the sig
nifican
ceo
f the resu
lts, we su
gg
ested th
at experim
ents th
at have
fewer co
un
ts than
expected
backg
rou
nd
also rep
ort th
eirsen
sitivity, w
hich
we d
efined
as the averag
e* u
pp
er limit th
atw
ou
ld b
e ob
tained
by an
ensem
ble o
f experim
ents w
ith th
eexp
ected b
ackgro
un
d an
d n
o tru
e sign
al. *Sh
ou
ld b
e med
ian
lW
e did
this in
the N
OM
AD
experim
ent an
d o
ther exp
erimen
tsh
ave been
do
ing
the sam
e thin
g.
Gary F
eldm
an N
EP
PS
R 23 A
ug
ust 2002 21
Visit to
Harvard
Statistician
s (1)
lT
ow
ards th
e end
of th
is wo
rk, I decid
ed to
try it ou
t on
som
ep
rofessio
nal statistician
s wh
om
I kno
w at H
arvard.
l T
hey to
ld m
e that th
is was th
e stand
ard m
etho
d o
f co
nstru
cting
a con
fiden
ce interval!
l I asked
them
if they co
uld
po
int to
a sing
le reference o
f an
yon
e usin
g th
is meth
od
befo
re.
l T
hey co
uld
no
t.
Gary F
eldm
an N
EP
PS
R 23 A
ug
ust 2002 22
Visit to
Harvard
Statistician
s (2)
lT
heir lo
gic:
lIn
statistical theo
ry there is a o
ne-to
-on
e corresp
on
den
ceb
etween
a hyp
oth
esis test and
a con
fiden
ce interval.
(Th
e con
fiden
ce interval is a h
ypo
thesis test fo
r eachvalu
e in th
e interval.)
lT
he N
eyman
-Pearso
n T
heo
rem states th
at the likelih
oo
dratio
gives th
e mo
st po
werfu
l hyp
oth
esis test.
lT
herefo
re, it mu
st be th
e stand
ard m
etho
d o
fco
nstru
cting
a con
fiden
ce interval.
Gary F
eldm
an N
EP
PS
R 23 A
ug
ust 2002 23
Ken
dall an
d S
tuart (1961)
lS
o I started
readin
g ab
ou
t hyp
oth
esis testing
.
lA
t the start o
f chap
ter 24 of K
end
all and
Stu
art’s Th
eA
dvan
ced T
heo
ry of S
tatistics (ch
apter 23 o
f Stu
art and
Ord
), I fou
nd
1 1/4 cryptic p
ages th
at pro
po
se this m
etho
dan
d its exten
sion
to erro
rs on
the b
ackgro
un
d.
lW
e were ab
le to in
clud
e a reference to
Ken
dall an
d S
tuart in
a no
te add
ed in
pro
of to
ou
r pap
er.
Gary F
eldm
an N
EP
PS
R 23 A
ug
ust 2002 24
Exten
sion
s
lT
his tech
niq
ue is m
ore g
eneral th
an th
e simp
le examp
lesd
escribed
here.
lT
he p
aper d
iscusses th
e app
lication
to n
eutrin
o o
scillation
s,in
wh
ich lim
its are set on
two
param
eters, sin22q an
d D
m2,
simu
ltaneo
usly.
lIt can
also b
e extend
ed to
cases in w
hich
the b
ackgro
un
ds
are no
t precisely kn
ow
n (b
ut w
e have n
ot yet p
ub
lished
this).
lIn
fact, I have yet to
find
a pro
blem
in th
e con
structio
n o
fclassical co
nfid
ence in
tervals and
regio
ns th
at is no
tso
lvable b
y the o
rderin
g p
rincip
le sug
gested
here.