Bayesian Reasoning
A/Prof Geraint Lewis A/Prof Peter Tuthill
Thomas Bayes (1702-1761) Pierre-Simon Laplace
(1749-1827)
“Probability theory is nothing but common sense, reduced to calculation.”
Laplace
Are you a Bayesian or Frequentist?
“There are 3 kinds of lies: Lies, Damned Lies, and Statistics”Benjamin Disraeli ...and Bayesian Statistics
4
Frequentists
Fig 1. A Frequentist Statistician Fig 2. Bayesian Statistics Conference
What is Inference?If A is true then B is true (Major Premise)
A is true (Minor Premise) therefore B is true (conclusion)
Deductive Inference (Logic) Aristotle 4th Century B.C.
B is False (Minor Premise) therefore A is False (conclusion)
} STRONG SYLLOGISMS
Inductive Inference (Plausible Reasoning)
A is false (Minor Premise) therefore B is less plausible
B is true (Minor Premise) therefore A is more plausible }WEAK
SYLLOGISMS
A = A,B (in Boolean notation)
A BT → T
F ← F
F → f t ← T
What is Inference?
Cause Effects or outcomes
Effects or observations
Possible Causes
Deductive Logic:
Inductive Logic:
What is a Probability?BayesiansFrequentists
P(A) = long run relative frequency of A occurring in identical repeats of an observation
“A” is restricted to propositions about random variables
P(A|B) = Real number measure of the plausibility of proposition A, given (conditional upon) the truth of proposition B
“A” can be any logical propositionAll probabilities are conditional; we must be explicit what our assumptions B are (no such thing as an absolute probability!)
Probability depends on our state of Knowledge
?
B CA
Monte Hall
The Desiderata of Bayesian Probability Theory
• Degrees of plausibility are represented by real numbers (higher degree of belief represented by a larger number)
• With extra evidence supporting a proposition, the plausibility should increase monotonically up to a limit (certainty).
• Consistency. Multiple ways to arrive at a conclusion must all produce the same answer (see book for additional details)
Logic and Probability
• In the certainty limit, where probabilities go to zero (falsehood) or one (truth), then the sum and product rules reduce to formal Boolean deductive logic (strong syllogisms).
• Bayesian Probability is therefore an extension of formal logic into intermediate states of knowledge.
• Bayesian inference gives a measure of our state of knowledge about nature, not a measure of nature itself.
The two rules underlyingprobability theory
SUM RULE: P(A|B) + P(A|B) = 1
PRODUCT RULE: P(A,B|C) = P(A|C) P(B|A,C) = P(B|C) P(A|B,C)
Left Handed
Right Handed
Blue Eyes
Brown Eyes All Kangaroos
Blue, Left
Bayes’ Theorem
Bayes Theorem: P(Hi|D,I) = P(D|I)
P(Hi|I) P(D|Hi I)
Hi = proposition asserting truth of a hypothesis of interest
I = proposition representing prior information
D = proposition representing the data
P(D|Hi I) = Likelihood: probability of obtaining the data given that the hypothesis is true
P(Hi|I) = Prior: probability of hypothesis before new data
P(D|I) = Normalization factor (prob all hypothesis i sum to 1)
Posterior
Example: The Gambler’s coin problem
P(H|D,I) = P(D|I)
P(H|I) P(D|H I)
Likelihood – if we assume the data D gives R heads in N tosses:
Prior – what do we know about the coin? Normalization factor – Ignore this for now as only need relative merit
Assume H=pdf(head) is uniformly distributed 0-1
P(D|H I) HR (1-H)N-R The full distribution, assuming independence of throws, is the Binomial Distribution. We omit terms not containing H, and use a proportionality.
Top Related