ECE 471/571 – Lecture 2

Bayesian Decision Theory08/25/15

ECE471/571, Hairong Qi 2

Different Approaches - More Detail

Pattern Classification

Statistical Approach Non-Statistical Approach

Supervised Unsupervised

Basic concepts: Distance Agglomerative method

Basic concepts: Baysian decision rule (MPP, LR, Discri.)

Parametric learning (ML, BL)

Non-Parametric learning (kNN)

NN (Perceptron, BP)

k-means

Winner-take-all

Kohonen maps

Dimensionality Reduction Fisher’s linear discriminant K-L transform (PCA)

Performance Evaluation ROC curve TP, TN, FN, FP

Stochastic Methods local optimization (GD) global optimization (SA, GA)

3

Bayes formula (Bayes rule)

a-posteriori probability(posterior probability)

a-priori probability(prior probability)

Conditional probability density(pdf – probability density function)(likelihood)

normalization constant(evidence)

From domain knowledge

4

pdf examples

Gaussian distribution Bell curve Normal distribution

Uniform distributionRayleigh distribution

5

Bayes decision rule

xMaximum a-posteriori Probability (MAP):

6

Conditional probability of error

7

Decision regions

The effect of any decision rule is to partition the feature space into c decision regions

8

Overall probability of error

Or unconditional risk, unconditional probability of error

8

9

The conditional risk

Given x, the conditional risk of taking action i is:

ij is the loss when decide x belongs to class i while it should be j

ij

10

Likelihood ratio - two category classification

Likelihood ratio

11

Zero-One loss

12

Recap

Bayes decision rule maximum a-posteriori probabilityConditional risk likelihood ratioDecision regions How to calculate the overall probability of error

13

Recap

Maximuma-posterioriProbability

Likelihood ratio

Overall probability of error