Pattern Recognition :

Feature Representation x

Decision Function d(x)Geometric Interpretation :

Decision Boundary

Neural NetworkConventional :

Statistical Formulation

- Bayes : Optimal

Syntactic ApproachMcCulloch–Pitts

Neuronal Model :

Threshold Logic

Sigmoid

Neuronal Model : Chap 3

Digital LogicSingle Layer

Network

Multilayer

Network

Geometric interpretation

(Surface)

Ch1 –

Concept Map

→ Can represent any region in space

Chapter 1. Pattern Recognition and Neural Networks

1. Pattern Recognition

Two ObjectivesClass Recognition : Image -> Apple

Attribute Recognition – Image -> Color, Shape, Taste, etc.

(1) Approachesa. Template Matchingb. Statistical c. Syntactic

d. Neural Network

Ex. Color (attribute) of apple is red.

Class 2Class 1

)( 2 x ωP)( 1 xω P

Statistical

x

Bayes Optimal Decision Boundary

Ex. x = temperature, ω1: healthy, ω2 : sick. x = (height, weight), ω1: female, ω2 : male.

(2) Procedure - Train and Generalize

Preprocessing Feature Extraction Discriminant Decision making ClassRaw

Data

For data reduction,better separation

x )(xd

Eliminate bad data (outliers)Filter out noise

Training Data = Labelled Input/Output Data = { x | d(x) is known }

(3) Decision ( Discriminant ) Function

a. 2-Class Weather Forecast Problem n = 2, M = 2

32211)( wxwxwd x

32211 wxwxw

unnormalized

normalized

20)( xxd 10)( xxd

xwwx ~

11

)( T2

1T

2

1

321

x

x

Dx

x

d o

0)( xd

1x

2x

12

decision boundary = ( n-1) dim. = line, plane, hyperplane.

=

22

21

32211

ww

wxwxw

w1 w2 w3

x1 = temp, x2 = pressure

.

. ,

.

.2

1

2

1

nn

o

x

x

x

w

w

w

xw

Dd o xwx T)(

Dn 1

In general,

xw T

0

D xw T

00

0T

0xw

D

x

0w

Dxw T

0

0T

0

T

0

xw

xw D

ow is a unit normal to the Hyperplane.

b. Case of M = 3 , n = 2 – requires 3 discriminants

IR1 2d

3 1d

3 1d2 3d

1

3

2

Pairwise Separable- Discriminants Req.

1

2

3

12d 23d 31d

+ 

-

- + 

 

- +

2CM

133 1 ddd

322 3 ddd

211 2 ddd

1

3

2

1d 2d 3d

1

2

3

Max   

 

Max 

   

Max

Linear classifier Machinexwxwpxg T

iii )|()( cci xxgxg )()(max

01 d

023 d

13

2

IR

1d

d2

3d

1 3

2

2. PR – Neural Network Representation

(1) Models of a Neuron

A. McCulloch-Pitts Neuron – threshold logic with fixed weights

1x

2x

px

1w

2w

pw

1x

2x

px

1

= bias

y

NonlinearActivation Function

Adaptive LinearCombiner

(Adaline)

ii

ii

xw

xwy

0

1

))(xdy

ii xwxd )(

B. Generalized Model

Two-Sided

One-Sided

Hard Limiter, Threshold Logic

1x

2x

+

1

2

bias

Half Plane Detector

0θxx 2211

1x

2x

1

2

bias

?

Ramp Logistic

Signum, Threshold Logic Ramp tanh

Binary

Bipolar

Piecewise Linear

(2) Boolean Function Representation Examples

1x

2x

AND1.5

1x INVERTER-0.5-1

MEMORY0.5-1

11Excitatory

Inhibitory

1x

2x

NAND-1.5

-1

-1

1x

2x

OR0.5

1x

2x

NOR-0.5`

-1

-1

x , x = binary (0, 1)1 2

Cf. x , x may be bipolar ( -1, 1) → Different Bias will be needed above.1 2

1x

2x

321

1 0 0

0 1 0

0 0 1

A. Single Layer with Single Output = Fire within a Half Plane for a 2-Class Case

(3) Geometrical Interpretation

B. Single Layer Multiple Outputs – for a Multi-Class Case

Linearly Nonseparable

0

-5

3

1

x 1

x2

x1 x 2

Nonlinearly Separable

C. Multilayer with Single Output - XOR

ON OFF

→ Bipolar Inputs,

Other weights needed for Binary Representation.

1x

2x

XOR

0.5

0.5

0.5

- 1

- 1

11

a. Successive Transforms

OR

21 xx

21 xx

1x

2x )( 21 xx

)(21

xx

21x =0.5 x

21x =0.5 x

1x

2x

XOR

-1.5

0.5

1.5

- 1

1

1- 1

1

1

AND

OR

NAND

1x

2x

0.5

1.5

0.5

1

1

111-bit parity

1x

2x

0.5

1.5

0.5 n-bit parity

nx n-0.5

1

-1

1

-1

(-1) n+1

b. XOR = OR ─ (1,1) AND

c. Parity

1

1

1.5 0.5-2

1

1

1x

2x

ORAND

= XOR

D. Multilayer with Single Output – Analog Inputs (1/2)

1

2

1

2

3

OR OR

1

2

1

2 3

1

2

3

1

2

3

AND

E. Multilayer Single Output – Analog Inputs – (2/2)

1 2

3

4

5

6

1

23

4

56

1

2

3AND

OR

4

5

6

1

2

3

4

5

6

AND

AND

OR

F.

23年 4月 20日 27

MLP Decision BoundariesMLP Decision Boundaries

A B

B A

A

B

XOR Interwined General

1-layer: Half planes

A B

B A

A

B

2-layer: Convex

A B

B A

A

B

3-layer: Arbitrary

0 1 x

①

x

1

2

1 ②

0 1 x

x

1

2

1

0 1 x

①

x

1

2

1

②

2

Exercise :Transform of NN from to : See how the weights are changed.① ②

0 1 2 3 x

②①

x

1

2

1

Questions from Students -05

How to learn the weights ? Any analytic and systematic

methodology to find the weights ? Why do we use polygons to

represent the active regions [1 output] ?

Why should di(x) be the maximum for the Class i ?