Hopefully a clearer version of Neural Network

I1

O2

O1H1

H2I2

Layers of Weights

• We Name Sets of Weights between layersAs W1 for weights between input Layer and

First Hidden LayerW2 for weights between next 2 layers and WN-1 for Weights between N-1th and Nth

Layer(i.e. Output Layer)In our example Net we just have 3 layersInput Hidden and Output So we have just

W1 and W2

I1

O2

O1H1

H2I2

W1W2

Weights along Individual Links

• Convention

• Each Weight is named as follows

• WNij

• N refers to the Layer of Weights

• So Between Input and First Hiden Layer i.e. W2ij is the Reference

• Between Hidden and Output W2ij

Individual Weights within a layer

• Reference WNij

• WN refers to the Weight Layer

• ij refers to the indices of the source and destination nodes.

• So for example the weight between hidden node h1 and output node o2

• It belongs to weight layer 2 so W2

• i = 1 and j = 2 so Weight is W212

I1

O2

O1H1

H2I2

W1W2

W212

Full Naming of Weight Set

I1

O2

O1H1

H2I2

W1 W2

W212W112

W221

W211

W222

W121

W111

W122

With Actual Weights

I1

O2

O1H1

H2I2

W1 W2

0-1

0

-1

-1

0

1

1

Inputs

• 1 and 0

• Target outputs {1,1}

I1

O2

O1H1

H2I2

W1 W2

0-1

0

-1

-1

0

1

1

1

0

Hidden Layer Computation

• Xi =iW1 = • 1 * 1 + 0 * -1 = 1, • 1 * -1 + 0 * 1 = -1 = • { 1 - 1} = {Xi1,Xi2} = Xi

xF

1

1

• h = F(X)• h1 = F(Xi1) = F(1)• h2 = F(Xi2) = F(-1)

27.01

1

1

1)2(

73.01

1

1

1)1(

)1(2

)1(1

xi

xi

XiF

XiF

I1

O2

O1H1

H2I2

W1 W2

0-1

0

-1

-1

0

1

1

1

0

0.73

0.27

Next Outputs

Output Layer Computation

• X = hW2 = • 0.73 * -1 + 0.27 * 0 = -0.73, • 0.73 * 0 + 0.27 * -1 = -0.27 =• { -0.73 - 0.27} = {X1,X2} = X

xF

1

1

• O = F(X)• O1 = F(X1)• O2 = F(X2)

433.01

1

1

1)2(

325.01

1

1

1)1(

)27.0(2

)73.0(1

x

x

XF

XF

I1

O2

O1H1

H2I2

W1 W2

0-1

0

-1

-1

0

1

1

1

0

0.73

0.27

0.325

0.433

Error

• D= Output(1 – Output)(Target – Output)• Target T1 = 1 , O1 = 0.325 = 0.33

• d1 = 0.33( 1 -0.33)(1 -0.33 ) = 0.33 (0.67)(0.67) = 0.148

• Target T2 = 1 , O2 = 0.433 = 0.43

• d2 = 0.43(1 - 0.43)(1-0.43) = 0.43(0.57)(0.57) = 0.14

Weight Adjustment

• △W2t = α hd + Θ △W2t-1

• where α = 1• Time t = 1 so no previous time

2212

211121

2

1

dhdh

dhdhdd

h

hhd

)14.0*27.0()15.0*27.0(

)14.0*73.0()15.0*73.0(14.015.0

27.0

73.0hd

Weight Adjustments

)038.0()04.0(

)102.0()109.0(

Weight Change

)038.02()04.02(

)102.02()109.02(

2221

1211

WW

WW

Equals

)038.01()04.00(

)102.00()109.01(

Equals

)962.0()04.0(

)102.0()891.0(

Putting these new weights in the diagram

• To get

I1

O2

O1H1

H2I2

W1 W2

0.102-1

0.04

-0.891

-0.962

0

1

1

Next

• Calculate Change on W1 layer weights

Error Calculatione = h(1 - h)W2d

21

11

h

h

2

1

d

d

2221

1211

22

22

WW

WW 21 hh

2

1

e

e

Another Way to write the error

outputsk

kikiih dWhhe )1(

What is this

• Outputs are O1 and O2

• So k = {1,2}• So if i = 1

outputsk

kikdW

summation

2121112,1,1

dWdWdWdWki

kikoutputsk

kik

I1

O2

O1H1

H2I2

W1 W2

0.102-1

0.04

-0.891

-0.962

0

1

1

This equals

• e1 = (h1(1-h1)W11 D1 +W12D2• e2 = (h2(1-h2)) W21 D1 +W22D2• d1 = 0.15 d2 = = 0.14e1 = (0.73(1-0.73))( -1* 0.15 +0*0.14)• e2 =( 0.27(1-0.27)) (0 *0.15 +-1*0.14)

• e1 = (0.73(0.27)( -0.15))• e2 =( 0.27(0.73)) (-0.14)• e1 = -0.03• e2 = -0.028

Weight Adjustment

• △W1t = α Ie + Θ △W2t-1

• where α = 1

2212

211121

2

1

eIeI

eIeIee

I

IIe

)028.0*0()03.0*0(

)028.0*1()03.0*1(028.003.0

0

1Ie

Weight Adjustment

)0()0(

)028.0()03.0(

Existing W1

10

11

11

11

2221

1211

WW

WW

Weight Change W1

)0(1)0(0

)028.0(1)03.0(1

New W1

10

028.197.0

• Changing Net

I1

O2

O1H1

H2I2

W1 W2

-0.102-1.028

-0.04

-1.109

-1.038

0

0.97

1