Back Propagation Learning Algorithm
description
Transcript of Back Propagation Learning Algorithm
Lecture 5
Introduction to Neural Networksand Fuzzy Logic
President University Erwin Sitompul NNFL 5/1
Dr.-Ing. Erwin SitompulPresident University
http://zitompul.wordpress.com
2 0 1 4
President University Erwin Sitompul NNFL 5/2
Back Propagation Learning AlgorithmNeural Networks
Backwardpropagation
f(.)
f(.)
f(.)
• Set the weights• Calculate output
1
1
( )( ) ( ) ( )
pl l lk k jl
ikj
Ei f net i y i
w
w
1 21
1
( )( ) ( ) ( ) ( )
pl l l l lk k kj j il
iji
Ei f net i w f net i y i
w
w
Forwardpropagation
1( ), ( )l lj ky i y i
• Calculate error• Calculate gradient
vector
• Update the weights
1
1
( ) ( )
( ) ( )
l lk k
ml l lk kj j
j
y i f net i
net i w x i
1 1
1 1 2
1
( ) ( )
( ) ( )
l lj j
nl l lj ji i
i
y i f net i
net i w y i
1
( ) ( ),
l lkj ji
E E
w w
w w
( )lk i
Multi Layer Perceptrons
President University Erwin Sitompul NNFL 5/3
Learning with Momentum In an effort to speed up the learning process, a weight
update is made to be based on the previous weight update. This is called momentum, because it tends to keep the error rolls down the error surface.
Because many updates to a particular weight are in the same direction, adding momentum will typically result in a speed up of learning time in many applications.
Multi Layer PerceptronsNeural Networks
When using momentum, the update rule for the network weights is modified to be:
( )( ) ( 1)kj kj
kj
Ew n w n
w
w
where α (typically 0.05) is the momentum and n is the number of iteration.
President University Erwin Sitompul NNFL 5/4
Learning with MomentumNeural Networks Multi Layer Perceptrons
The momentum can be seen to be practically increasing the learning rate.
This is in accordance with several heuristics that should be used in neural network design to improve the result:Each weight should have a local learning rateEach learning rate should be allowed to vary over
timeConsecutive updates with the same sign to a weight
should increase the weights’ learning rateConsecutive updates with alternating sign to a
weight should decreases the weights’ learning rate.
President University Erwin Sitompul NNFL 5/5
Learning with Weighted MomentumNeural Networks Multi Layer Perceptrons
Various efforts have focused on deriving additional forms of momentum.
One such method is to relate the momentum of the previous update and the current calculation of error gradient.
By doing so, the momentum can be preserved when an iteration attempts to update contrary to the most recent updates.
The update rule for the network weights in this case is given by:
( )( ) (1 ) ( 1)kj kj
kj
Ew n w n
w
w
President University Erwin Sitompul NNFL 5/6
Learning with Variable Learning RateNeural Networks Multi Layer Perceptrons
The basic idea: speed up the convergence by increasing the learning rate on flat surface and decreasing it when the slop increases.
If error increases by more than a predefined percentage θ (i.e. 1-5%) then: Weight update is discarded Learning rate is decreased by a factor 0< γ <1, i.e. γ = 0.7 Set momentum to zero
If error increases by less than θ: Weight update is accepted Learning rate is unchanged If momentum has been set to zero, it is reset to original value
If the error decreases: Weight update is accepted Learning rate is increased by some factor β >1, i.e. β = 1.05 If momentum has been set to zero, it is reset to its original
value
President University Erwin Sitompul NNFL 5/7
Feedforward Network
InputNeuronLayer
NeuronLayer
Output
f(.)
f(.)
f(.)
MLP for System ModelingNeural Networks
President University Erwin Sitompul NNFL 5/8
Feedforward Network
01 2 1 0 2 0y
02 3 1 3 1 0y
21 17 3 9 9 0d
01 ( )y i
02 ( )y i
21 ( )y i
MLP for System ModelingNeural Networks
President University Erwin Sitompul NNFL 5/9
Recurrent NetworksExternal Recurrence
Internal Recurrence
Input NeuronLayer
NeuronLayer
Output
Time Delay
Element
Time Delay
Element
Input NeuronLayer
NeuronLayer
Output
Time Delay
Element
MLP for System ModelingNeural Networks
President University Erwin Sitompul NNFL 5/10
InputDynamicSystem
Output
( )u k ( )y k
Dynamic System
( ) ( , )y k m g
a b( 1), , ( ), ( 1), , ( )y k y k n u k u k n g
System parameter
Input-output data vector
MLP for System ModelingNeural Networks
President University Erwin Sitompul NNFL 5/11
InputDynamic
Model
Output
( )u k ˆ( )y k
Dynamic Model
ˆ( ) ( , , )y k w b g
a b( 1), , ( ), ( 1), , ( )y k y k n u k u k n g
weightsbias
input-output data vector
MLP for System ModelingNeural Networks
President University Erwin Sitompul NNFL 5/12
Neural Network Dynamic Model
Feedforward
ˆ( )y k : model output,estimate of system output
( )y k : system output. . .
. . .
. . .
ˆ( )y k
. . .
( 1)u k
b( )u k n
( 1)y k
a( )y k n
MLP for System ModelingNeural Networks
President University Erwin Sitompul NNFL 5/13
Neural Network Dynamic Model
Recurrent
. . .
. . .
. . .
ˆ( )y k
. . .
( )u k
1z
anz
1z
bnz
MLP for System ModelingNeural Networks
President University Erwin Sitompul NNFL 5/14
Tapped Delay Line (TDL)
( )u k
( 1)u k ( 2)u k
( 3)u k ( )u k n
1z 1z 1z 1z .....
( )u k
( 1)u k ( )u k n
T D L
.....
MLP for System ModelingNeural Networks
Unit 1 Unit 2 Unit 3 Unit n
President University Erwin Sitompul NNFL 5/15
Implementation
InputDynamicSystem
Output
( )u k ( )y k
ˆ( )y k. . .
. . .
T D L T D L
feedforward
external recurrence
MLP for System ModelingNeural Networks
President University Erwin Sitompul NNFL 5/16
ExampleSingle Tank System
2
20.4 m0.012 m
Aa
outq
inq
hLearning Data Generation
A : cross-sectional area of the tanka : cross-sectional area of the pipe
Area of operation
Save data to workspace
MLP for System ModelingNeural Networks
in a
1 ah q v
A A
in
12
ah q gh
A A
President University Erwin Sitompul NNFL 5/17
Example
( 1)u k ( )y k
( 1)y k
Data size : 201 from 200 seconds of
simulation
0 20 40 60 80 100 120 140 160 180 2000
0.02
0.04
0.06
0.08
0.1
0.12
0 20 40 60 80 100 120 140 160 180 2000
0.02
0.04
0.06
0.08
0.1
0.12
Feedforward Network External Recurrent Network
MLP for System ModelingNeural Networks
2–2–1 Network
President University Erwin Sitompul NNFL 5/18
Homework 5
( 1)u k
y k( 2)u k
( 1)y k
( 2)y k
0 20 40 60 80 100 120 140 160 180 200-0.025
-0.02
-0.015
-0.01
-0.005
0
0.005
0.01
0.015
0.02
Delta of 2–2–1 network
4–4–1 Network
MLP for System ModelingNeural Networks
A neural network with 2 inputs and 2 hidden neurons seems not to be good enough to model the Single Tank System. Now, design a neural network with 4 inputs and 4 hidden neurons to model the system. Use bias in all neurons and take all a = 1.
Be sure to obtain decreasing errors.
Submit the hardcopy and softcopy of the m-file.
President University Erwin Sitompul NNFL 5/19
Homework 5A (smaller Student-ID)
( 1)u k
y k( 3)u k
( 1)y k
( 4)y k 4–4–1 Network
MLP for System ModelingNeural Networks
For 4 students with smaller Student ID, redo Homework 5 with the following changes:The network inputs are u(k–1), u(k–3), y(k–1), and y(k–4)
The activation functions for the neurons in the hidden layer are:
Be sure to obtain decreasing errors.
Compare the result with the previous result of HW5.
Submit the hardcopy and softcopy of the m-file.
President University Erwin Sitompul NNFL 5/20
Homework 5B (greater Student-ID)MLP for System ModelingNeural Networks
Be sure to obtain decreasing errors.
Compare the result with the previous result of HW5.
Submit the hardcopy and softcopy of the m-file.
( 1)u k
( )y k( 1)y k
( 2)y k
( 3)y k 4–4–1 Network
For 4 students with greater Student ID, redo Homework 5 with the following changes:The network inputs are u(k–1), y(k–1), y(k–2), and y(k–3)
The activation functions for the neurons in the hidden layer are: