Everything You Always Wanted To Know About
Limits*Roger Barlow
Manchester UniversityYETI06
*But were afraid to ask
Roger Barlow: YETI06 Everything you wanted to know about limits
Slide 2
SummaryPrediction
confronts data & sees
small/zero signal
Frequentist
probability and
Confidence
Level languageBayesian
Probability
(Health
Warning)Gaussian
ln L= -½
Zero events
Few events:
Confidence belt The horrendous
case of large
backgrounds
Extension to
several
parameters
Likelihood
Roger Barlow: YETI06 Everything you wanted to know about limits
Slide 3
Model predictionsInput model and parameters
Low energyLagrangian
Feynman Rules for Feynman
diagrams
Cross Sectionsand Branching
RatiosExperiment duration,
luminosity,Efficiency etc
Number of events
Monte Carloprograms
Cuts designed tobring out signal
Data
Roger Barlow: YETI06 Everything you wanted to know about limits
Slide 4
What happens if there’s nothing there?
Even if your analysis finds no events, this is still useful information about the way the universe is built
Want to say more than: “We looked for X, we didn’t see it.”
Need statistics – which can’t prove anything.
“We show that X probably has a mass greater than../a coupling smaller than…”
Roger Barlow: YETI06 Everything you wanted to know about limits
Slide 5
Probability(1): Frequentist
Define Probability of X as P(X)=Limit N∞ N(X)/N
Examples: coins, dice, cards For continuous x extend to Probability
DensityP(x to x+dx)=p(x)dx
Examples: • Measuring continuous quantities (p(x)
often Gaussian)• Parton momentum fractions (proton pdfs)
Roger Barlow: YETI06 Everything you wanted to know about limits
Slide 6
Digression: likelihood
Probability distribution of random variable x often depends on some parameter a.
Joint function p(x,a)Considered as p(x)|a this is the pdf.
Normalised: ∫p(x)dx=1Considered as p(a)|x this is the Likelihood L(a)Not ‘likelihood of a’ but ‘likelihood that a
would give x’Not normalised. Indeed, must never be
integrated.
Roger Barlow: YETI06 Everything you wanted to know about limits
Slide 7
Limitation of Frequentist Probability
Have to say“The statement ‘It will rain tomorrow.’ is
probably true.”Can then even quantify (meteorology).
Can’t say“It will probably rain tomorrow.”
There is only one tomorrow. P is either 1 or 0
Roger Barlow: YETI06 Everything you wanted to know about limits
Slide 8
Interpreting physics results: Mt =173±2 GeV/c2
Can’t say ‘Mt has a 68% probability of lying between 171
and 175 GeV/c2’Have to say‘The statement “Mt lies between 171 and 175
GeV/c2”has a 68% probability of being true’i.e. if you always say a value lies within its error
bars, you will be right 68% of the time.Say “Mt lies between 171 and 175 GeV/c2” with
68% Confidence. Or 169-177 with 95% confidence. Or…
Roger Barlow: YETI06 Everything you wanted to know about limits
Slide 9
Interpreting null resultYour analysis searches for events. Sees none.Use Poisson formula: P(n; )=e-n/n!Small could well give 0 events =0.5 gives P(0)=61% =1.0 gives P(0)=37% =2.3 gives P(0)=10% =3.0 gives P(0)=5%If you always say ‘ 3.0’ you will be right (at least)
95% of the time. 3.0 – with 95% confidence (a.k.a 5% significance.)
‘If is actually 3, or more, the probability of a fluctuation as far as zero is only 5%, or less.’
given by model parameters. Limit on translates to limit on mass, coupling, ,branching ratio or whatever
Roger Barlow: YETI06 Everything you wanted to know about limits
Slide 10
Probability(2): Bayesian
P(X) expresses by degree of belief in XCan calibrate against cards, dice, etc.Extend to probability density p(x) as
beforeNo restrictions on X or x. Rain, MT, MH,
whateverInterpret physics results using Bayes’
Theorem:pposterior(a|data) p(data|a) x pprior(a)
Roger Barlow: YETI06 Everything you wanted to know about limits
Slide 11
Bayes at work
= x
P(0 events|)
(Likelihood)
Prior: uniformPosterior P()
3 P() d= 0.95
0
Same as Frequentist limit - Happy coincidence
Zero events seen
Roger Barlow: YETI06 Everything you wanted to know about limits
Slide 12
Bayes at work again
= x
P(0 events|) Prior: uniform in ln Posterior P()
3 P() d >> 0.95
0
Is that uniform prior really credible?
Upper limit totally different!
Roger Barlow: YETI06 Everything you wanted to know about limits
Slide 13
Bayes: the bad news• The prior affects the posterior. It is your choice. That
makes the measurement subjective. This is BAD. (We’re physicists, dammit!)
• A Uniform Prior does not get you out of this.• SUSY ‘parameter space’ is not a ‘phase space’• Attempts to invent universally-agreed priors
(‘Objective’ and/or ‘Reference’ Priors) have not worked
Better news: If there is a lot of data then the prejudicial effects of the choice of prior can be small.
• This should ALWAYS be checked for (‘robustness under choice of prior’.)
Roger Barlow: YETI06 Everything you wanted to know about limits
Slide 14
Frequentist versus Bayesian?
Statisticians do a lot of work with Bayesian statistics and there are a lot of useful ideas. But they are careful about checking for robustness under choice of prior.
Beware snake-oil merchants in the physics community who will sell you Bayesian statistics (new – cool – easy – intuitive) and don’t bother about robustness.
Use Frequentist methods when you can and Bayesian when you can’t (and check for robustness.) But ALWAYS be aware which you are using.
Roger Barlow: YETI06 Everything you wanted to know about limits
Slide 15
A Gaussian MeasurementNo problems
p(x)=exp[-(x-)2/2 2]/√2x: symmetric
x is within ± of with 68% probability is within ± of x at 68% confidencex is above -1.645 with 95% probability is below x+1.645 at 95% confidenceChoice of confidence level and arrangement
Can read regions off log likelihood plot asL(a)=exp[-(x-)2/2 2]/√2
Ln L -(-x)2/2 2
68% region corresponds to fall of ½ from peak
Roger Barlow: YETI06 Everything you wanted to know about limits
Slide 16
A Poisson measurement
You detect 5 events. Best value 5. But what about the errors?
1. 5±√5=5±2.24 Assumes e-n/n! is Gaussian in n. True only for large - and 5 is small
2. Find points where log likelihood falls by ½.
Assumes e-n/n! is Gaussian in .Gives upper error of 2.58, lower error of 1.92
Roger Barlow: YETI06 Everything you wanted to know about limits
Slide 17
3: Doing it properly: Confidence belt (Neyman
interval)Use e-n/n! For any true the
probability that (n, ) is within the belt is 68% (or more) by construction
For any n, lies in [-, +] at 68% confidence
Get upper error 3.38, lower error 2.16
n
-
+
68%
16%16%
Technique works for any CL, and single or double sided
Roger Barlow: YETI06 Everything you wanted to know about limits
Slide 18
Consumer guide
ln L =- ½ is a standard and easy to use. Fine for everyday use. (Though for a simple count the Neyman limit is quite easy)
For 90% 1-sided (upper) limit use ln L =-0.82 (1.28 ) For 95% use ln L =-1.35 (1.645 ) Just plot the likelihood and read off the
value. Then translate back to model parameters
Roger Barlow: YETI06 Everything you wanted to know about limits
Slide 19
Frequency method: the big problem
Observe 5 events. Expected background of 0.9 events.Data = signal + background
Say with 68% confidence: data in range 2.84 to 8.38So say with 68% confidence: signal in range 1.94 to
7.48Suppose expected* background 4.9. Or 6.9. Or 10.9 ?“We say at 68% confidence that the number of signal
events lies between -8.06 and -2.52”This is technically correct. We are allowed to be wrong
32% of the time. But stupid. We know that the background happens to have a downward fluctuation but have no way of incorporating that knowledge
*We assume that the background is calculated correctly
Roger Barlow: YETI06 Everything you wanted to know about limits
Slide 20
Strategy 1: Bayes
Prior is uniform for positive , zero for negative . No problem.
Get requirement (for n observed, known background b, 90% upper limit)
0.1=nexp(-+-b) (++b)r/r!
nexp(-b) br/r!Known as “the old PDG formula” or
“Helene’s formula” or “that heap of crap”
Roger Barlow: YETI06 Everything you wanted to know about limits
Slide 21
Strategy 2: Feldman-Cousins
Also called* ‘the Unified Approach’Real physicists wait to see their result and then
decide whether to quote an upper limit or a range.This ‘flip-flopping’ invalidates the method.They provide a procedure that incorporates it
automatically, and always gives non-stupid results.Critics say (1) can lead to experiments quoting a
range when they’re not claiming a discovery (2) is computationally intensive and (3) For zero observed events, the higher the background estimate the better (i.e. lower) the limit on signal
* By Feldman and Cousins, principally
Roger Barlow: YETI06 Everything you wanted to know about limits
Slide 22
Strategy 3: CLs
As used by LEP Higgs working groupGeneralisation of Helene formulaSome quantity Q. Could be number of
events, or something more cleverCLb=P(Q or less|b) CLs+b=P(Q or less|s+b)
CLs=CLs+b/CLb
Used as confidence level. Optimise strategy using it and quote results
Roger Barlow: YETI06 Everything you wanted to know about limits
Slide 23
2(+) parameters Fix b, find 68% confidence
range for a, using ln L=-½
Fix a, find 68% range for bCombination (square) has
0.682=46%
a
b
L(a,b)
ln L=-½ circle has 39% ConfidenceDefine regions through contours of log L – Confidence
content given by 2ln L= for which P(n)=CL Caution! Cannot redefine a as b+c+d, claim 3
parameters and cut with P(3) instead of P(1)
Roger Barlow: YETI06 Everything you wanted to know about limits
Slide 24
SummaryPrediction
confronts data & sees
small/zero signal
Frequentist
probability and
Confidence
Level languageBayesian
Probability
(Health
Warning)Gaussian
ln L= -½
Zero events
Few events:
Confidence belt The horrendous
case of large
backgrounds
Extension to
several
parameters
Likelihood
Roger Barlow: YETI06 Everything you wanted to know about limits
Slide 25
Remember!
Zero events = 95% CL upper limit of 3 events
If it’s more involved, plot the likelihood function and use ln L=-½ for 68% central, etc
Be suspicious of anything you don’t understand
If you’re integrating the likelihood you are a Bayesian. I hope you know what you’re doing.
Roger Barlow: YETI06 Everything you wanted to know about limits
Slide 26
Further Reading
• Workshop on Confidence Limits, CERN yellow report 2000-005
• Proc. Conf. Advanced Statistical Techniques in Particle Physics, Durham, IPPP02/39
• Proc. PHYSTAT03 – SLAC-R-703• Proc PHYSTAT05, Oxford -
forthcoming
Top Related