INTRODUCTION
TO
MACHINE
LEARNING 3RD EDITION
ETHEM ALPAYDIN
© The MIT Press, 2014
http://www.cmpe.boun.edu.tr/~ethem/i2ml3e
Lecture Slides for
CHAPTER 3:
BAYESIAN DECISION
THEORY
Probability and Inference 3
Result of tossing a coin is {Heads,Tails}
Random var X {1,0}
Bernoulli: P {X=1} = poX (1 ‒ po)
(1 ‒ X)
Sample: X = {xt }Nt =1
Estimation: po = # {Heads}/#{Tosses} = ∑t xt / N
Prediction of next toss:
Heads if po > ½, Tails otherwise
Classification
Credit scoring: Inputs are income and savings.
Output is low-risk vs high-risk
Input: x = [x1,x2]T ,Output: C Î {0,1}
Prediction:
otherwise 0
)|()|( if 1 choose
or
otherwise 0
)|( if 1 choose
C
C
C
C
,xxCP ,xxCP
. ,xxCP
2121
21
01
501
4
Bayes’ Rule
x
xx
p
pPP
CCC
| |
110
0011
110
xx
xxx
||
||
CC
CCCC
CC
Pp
PpPpp
PP
5
posterior
likelihood prior
evidence
Bayes’ Rule: K>2 Classes
K
kkk
ii
iii
CPCp
CPCp
p
CPCpCP
1
|
|
||
x
x
x
xx
xx | max | if choose
and 1
kkii
K
iii
CPCPC
CPCP
10
6
Losses and Risks
Actions: αi
Loss of αi when the state is Ck : λik
Expected risk (Duda and Hart, 1973)
xx
xx
|min| if choose
||
kkii
k
K
kiki
RR
CPR
1
7
Losses and Risks: 0/1 Loss
ki
kiik
if
if
1
0
x
x
xx
|
|
||
i
ikk
K
kkiki
CP
CP
CPR
1
1
8
For minimum risk, choose the most probable class
Losses and Risks: Reject
10
1
1
0
otherwise
if
if
,Ki
ki
ik
xxx
xx
|||
||
iik
ki
K
kkK
CPCPR
CPR
1
1
1
otherwise reject
| and || if choose 1xxx ikii CPikCPCPC
9
Different Losses and Reject 10
Equal losses
Unequal losses
With reject
Discriminant Functions
Kigi ,, , 1x xx kkii ggC max if choose
xxx kkii gg max| R
ii
i
i
i
CPCp
CP
R
g
|
|
|
x
x
x
x
11
K decision regions R1,...,RK
K=2 Classes
Dichotomizer (K=2) vs Polychotomizer (K>2)
g(x) = g1(x) – g2(x)
Log odds:
otherwise
if choose
2
1 0
C
gC x
x
x
|
|log
2
1
CP
CP
12
Utility Theory
Prob of state k given exidence x: P (Sk|x)
Utility of αi when state is k: Uik
Expected utility:
xx
xx
| max| if Choose
||
jj
ii
kkiki
EUEUα
SPUEU
13
Association Rules
Association rule: X Y
People who buy/click/visit/enjoy X are also likely to
buy/click/visit/enjoy Y.
A rule implies association, not necessarily causation.
14
Association measures 15
Support (X Y):
Confidence (X Y):
Lift (X Y):
customers
and bought whocustomers
#
#,
YXYXP
X
YX
XP
YXPXYP
bought whocustomers
and bought whocustomers
|
#
#
)(
,
)(
)|(
)()(
,
YP
XYP
YPXP
YXP
Example 16
Apriori algorithm (Agrawal et al.,
1996) 17
For (X,Y,Z), a 3-item set, to be frequent (have
enough support), (X,Y), (X,Z), and (Y,Z) should be
frequent.
If (X,Y) is not frequent, none of its supersets can be
frequent.
Once we find the frequent k-item sets, we convert
them to rules: X, Y Z, ...
and X Y, Z, ...
Top Related