Classification Bayes
-
Upload
hussein-razaq -
Category
Documents
-
view
153 -
download
6
Transcript of Classification Bayes
March 2006 Alon Slapak 1 of 1
Bayes Classification
A practical approachExample
Discriminant function
Bayes theorem
Bayes discriminant
function
Bibliography
March 2006 Alon Slapak 2 of 2
Discriminant functionDefinition: a discriminant function is an n-dimensional hypersurface which divides the n-dimensional feature space into two separate areas contain separate classes.
A 2-dimwnsinaldiscriminant function
A 1-dimwnsinaldiscriminant function
2-dimensional feature space
1-dimensional feature space Example
Discriminant function
Bayes theorem
Bayes discriminant
function
Bibliography
March 2006 Alon Slapak 3 of 3
Discriminant functionLet h(x) be a discriminant function. A two-category classifier uses the following rule:
Decide 1 if h(x) > 0 and 2 if h(x) < 0
If h(x) = 0 x is assigned to either class.
h(x)=x-6.25h(x) < 0
h(x) < 0
1 2( ) 1.02 2.5h x x x
Example
Discriminant function
Bayes theorem
Bayes discriminant
function
Bibliography
March 2006 Alon Slapak 4 of 4
Thomas BayesAt the time of his death, Rev. Thomas Bayes (1702 –1761) left behind two unpublished essays attempting to determine the probabilities of causes from observed effects. Forwarded to the British Royal Society, the essays had little impact and were soon forgotten.
When several years later, the French mathematician Laplace independently rediscovered a very similar concept, the English scientists quickly reclaimed the ownership of what is now known as the “Bayes Theorem”.
Example
Discriminant function
Bayes theorem
Bayes discriminant
function
Bibliography
March 2006 Alon Slapak 5 of 5
Conditional ProbabilityDefinition: Let A and B be events with P(B) > 0.The conditional probability of A given B, denoted by P(A|B), is defined as:
P(A|B) = P(A B)/P(B)
A B
1020 5
Venn Diagram
Given: N(A) = 30
N(A B) = 10
P(B | A) = N(A B)/N(A) = 10/30 = 1/3
Example:Example
Discriminant function
Bayes theorem
Bayes discriminant
function
Bibliography
March 2006 Alon Slapak 6 of 6
Bayes’ theoremSince P(A | B) = P(A B)/P(B),
we have: P(A | B)P(B) = P(A B)
Symmetrically we have: P(B | A)P(A) = P(B A) = P(A B)
Therefore: P(A | B)P(B) = P(B | A)P(A)
And:
Bayes' theoremP A B P B
P B AP A
where P(A | B) is the conditional probability, P(A) , P(B) are the prior probabilities, P(B | A) is the posterior probability
Example
Discriminant function
Bayes theorem
Bayes discriminant
function
Bibliography
March 2006 Alon Slapak 7 of 7
Bayes’ theorem in a pattern recognition notation
Given classes i and a pattern x,
( | ) ( )( | )
( )
( ) ( | ) ( )
j jj
j jj
P PP
P
where
P P P
xx
x
x x
prior probability
• The prior probability reflects knowledge of the relative frequency of instances of a class
likelihood
• The likelihood is a measure of the probability that a measurement value occurs in a class.
evidence
• The evidence is a scaling term
Example
Discriminant function
Bayes theorem
Bayes discriminant
function
Bibliography
March 2006 Alon Slapak 8 of 8
Bayes classifierThe following phrase classify each pattern x to one of two classes:
2
1
2
1
1 2
1 1 2 2
( | ) ( | )
( | ) ( ) ( | ) ( )
( ) ( )
P P
P P P P
P P
x x
x x
x x
or (since P(x) is common to both sides):
2
1
2
1
1 1 2 2
1 2
2 1
( | ) ( ) ( | ) ( )
( | ) ( )
( | ) ( )
P P P P
P Pl
P P
x x
xx
x
Means, decide 1 if P(1|x) > P(2|x)
Likelihood ratio
Example
Discriminant function
Bayes theorem
Bayes discriminant
function
Bibliography
March 2006 Alon Slapak 9 of 9
Bayes Discriminant functionSince a ratio of probabilities may yield very small values, it is common to use the log of the likelihood ratio:
and the derived Bayes’ discriminant function is:
11 2
2
( )ln ( | ) ln ( | ) ln
( )
Ph P P
P
x x x
2
1
1 2
2 1
( | ) ( )ln ln ln
( | ) ( )
P Pl
P P
xx
x
Remember: Decide 1 if h(x) > 0 and 2 if h(x) < 0If h(x) = 0 x is assigned to either class.
Example
Discriminant function
Bayes theorem
Bayes discriminant
function
Bibliography
March 2006 Alon Slapak 10 of 10
Example - Gaussian Distributions
A multi dimensional Gaussian distribution is:
21
2/ 2 1/ 2
2 1
th
th
1
2 | |
where:
( ) ( )
is the feature vector
is the mean vector of the class
is the covariance matrix of the class
feature space dimension
i
i
d
i n
i
Ti i i
i
i
P e
d
i
i
n
xx
x x m x m
x
m
120 130 140 150 160 170 180 19030
35
40
45
50
55
60
65
70
75
80
height [cm]
wei
ght
[kg]
Females
Example
Discriminant function
Bayes theorem
Bayes discriminant
function
Bibliography
March 2006 Alon Slapak 11 of 11
Example - Gaussian Distributions
A multi dimensional Gausian distribution is:
2 21 2
1 112 2
/ 2 / 21/ 2 1/ 221 2
2 2 11 11 1 2 22 2 2 2
2
2 2 1 111 2 2
2 2
( )1 1ln ln ln
( )2 | | 2 | |
( )1 1ln 2 ln | | ln 2 ln | | ln
2 2 ( )
| | ( )1 1ln ln
2 2 | | ( )
d d
n n
n n
Pe e
P
Pd d
P
d
h
Pd
P
x x
x x
x
x
x
1 1 1 111 1 1 2 2 2 2
2 2
| | ( )1 1( ) ( ) ( ) ( ) ln ln
2 2 | | ( )T T P
P
x m x m x m x m
Example
Discriminant function
Bayes theorem
Bayes discriminant
function
Bibliography
March 2006 Alon Slapak 12 of 12
Example - Gaussian DistributionsAssume two Gaussian distributed classes withAnd
clear allN1 = 150; N2 = 150;E1 = [50 40; 40 50]; E2 = [50 40; 40 50];M1 = [30,55]'; M2 = [60,40]';%-------------------------------------------------------------------------% Classes drawing%-------------------------------------------------------------------------[P1,A1] = eig(E1); [P2,A2] = eig(E2);y1=randn(2,N1); y2=randn(2,N2);for i=1:N1, x1(:,i) =P1*sqrt(A1)* y1(:,i)+M1;end;for i=1:N2, x2(:,i) =P2*sqrt(A2)* y2(:,i)+M2;end;figure;plot(x1(1,:),x1(2,:),'^',x2(1,:),x2(2,:),'or');axis([0 100 0 100]);xlabel('x1')ylabel('x2')
1 2 1 2( ) ( )P P
2 1 1 1 2 2
1
2T T Th x m m x m m m m
Example
Discriminant function
Bayes theorem
Bayes discriminant
function
Bibliography
March 2006 Alon Slapak 13 of 13
Example - Gaussian Distributions%-------------------------------------------------------------------------% Classifier drawing%-------------------------------------------------------------------------ep=1.2;k=1;for i=0:k:100, for j=0:k:100, x=([i;j]); h=0.5*(x-M1)'*inv(E1)*(x-M1)-0.5*(x-M2)'*inv(E2)*(x-M2)+0.5*log(det(E1)/det(E2)); if (abs(h)<ep), hold on; plot(i,j,'*k'); hold off; end; end;end; h(x) > 0
h(x) < 0
2 1 1 1 2 2
10
2T T Th x m m x m m m m
Example
Discriminant function
Bayes theorem
Bayes discriminant
function
Bibliography
March 2006 Alon Slapak 14 of 14
Example - Gaussian DistributionsAssume two Gaussian distributed classes withAnd
clear allN1 = 150; N2 = 150;E1 = [50 40; 40 50]; E2 = [50 -40; -50 50];M1 = [30,55]'; M2 = [60,40];'
-------------------------------------------------------------------------% %Classes drawing
-------------------------------------------------------------------------%]P1,A1 = [eig(E1); [P2,A2] = eig(E2);
y1=randn(2,N1); y2=randn(2,N2);for i=1:N1,
x1(:,i) =P1*sqrt(A1)* y1(:,i)+M1;end;for i=1:N2,
x2(:,i) =P2*sqrt(A2)* y2(:,i)+M2;end;figure;plot(x1(1,:),x1(2,:),'^',x2(1,:),x2(2,:),'or');axis([0 100 0 100]);
xlabel('x1')ylabel('x2')
1 2 1 2( ) ( )P P
1 1 111 1 1 2 2 2 2
2
| |1 1( ) ( ) ( ) ( ) ln
2 2 | |T Th
x x m x m x m x m
Example
Discriminant function
Bayes theorem
Bayes discriminant
function
Bibliography
March 2006 Alon Slapak 15 of 15
Example - Gaussian Distributions%-------------------------------------------------------------------------% Classifier drawing%-------------------------------------------------------------------------ep=1;k=1;for i=0:k:100, for j=0:k:100, x=([i;j]); h=0.5*(x-M1)'*inv(E1)*(x-M1)-0.5*(x-M2)'*inv(E2)*(x-M2)+0.5*log(det(E1)/det(E2)); if (abs(h)<ep), hold on; plot(i,j,'*k'); hold off; end; end;end; h(x) > 0
h(x) < 0
h(x) < 0
11 11 1 1 2 2 2
2
1 1 1( ) ( ) ( ) ( ) ( ) ln 0
2 2 2T Th
x x m x m x m x m
Example
Discriminant function
Bayes theorem
Bayes discriminant
function
Bibliography
March 2006 Alon Slapak 16 of 16
Exercise• Synthesize two classes with different a-priory probabilities. Show how the probabilities influence the discriminant function.
• Synthesize three classes and plot the discriminant functions. Example
Discriminant function
Bayes theorem
Bayes discriminant
function
Bibliography
March 2006 Alon Slapak 17 of 17
Summary
Steps for Building a Bayesian Classifier
• Collect class exemplars
• Estimate class a priori probabilities
• Estimate class means
• Form covariance matrices, find the inverse and determinant for each
• Form the discriminant function for each classExample
Discriminant function
Bayes theorem
Bayes discriminant
function
Bibliography
March 2006 Alon Slapak 18 of 18
Bibliography1. K. Fukunaga, Introduction to Statistical Pattern
Recognition, 2nd ed., Academic Press, San Diego, 1990.
2. L. I. Kuncheva, J. C. Bezdek amd R. P.W. Duin, “Decision Templates for Multiple Classier Fusion: An Experimental Comparison”, Pattern Recognition, 34, (2), pp. 299-314, 2001.
3. R. O. Duda, P. E. Hart and D. G. Stork, Pattern Classification (2nd ed), John Wiley & Sons, 2000.
Example
Discriminant function
Bayes theorem
Bayes discriminant
function
Bibliography