Artificial Neural Networks (1)
description
Transcript of Artificial Neural Networks (1)
![Page 1: Artificial Neural Networks (1)](https://reader037.fdocuments.net/reader037/viewer/2022102819/5681432e550346895daf9f47/html5/thumbnails/1.jpg)
Artificial Neural Networks (1)
Dr. Hala Farouk
![Page 2: Artificial Neural Networks (1)](https://reader037.fdocuments.net/reader037/viewer/2022102819/5681432e550346895daf9f47/html5/thumbnails/2.jpg)
Biological Neurons of the Brain
![Page 3: Artificial Neural Networks (1)](https://reader037.fdocuments.net/reader037/viewer/2022102819/5681432e550346895daf9f47/html5/thumbnails/3.jpg)
History of Neural Networks
In 1943, Warren McCulloch and Walter Pitts introduced one of the first artificial neurons.
The main feature of their neuron model is that a weighted sum of input signals is compared to a threshold to determine the neuron output.
When the sum is greater than or equal to the threshold, the output is 0.
They went on to show that networks of these neurons could, in principle, compute any arithmetic or logical function.
Unlike biological networks, the parameters of their networks had to be designed, as no training method was available.
In the late 1950s, Frank Rosenblatt introduces a learning rule for training perceptron networks to solve pattern recognition problems. Perceptron learn from their mistakes even if they were initialized with random weights.
In the 1980s, improved multi-layer perceptron networks were introduced.
![Page 4: Artificial Neural Networks (1)](https://reader037.fdocuments.net/reader037/viewer/2022102819/5681432e550346895daf9f47/html5/thumbnails/4.jpg)
Analogy
![Page 5: Artificial Neural Networks (1)](https://reader037.fdocuments.net/reader037/viewer/2022102819/5681432e550346895daf9f47/html5/thumbnails/5.jpg)
Single-Input Neuron
The output depends on the transfer function f chosen by the designerW and b are parameters adjustable by some learning rule
Weight w
Bias b
![Page 6: Artificial Neural Networks (1)](https://reader037.fdocuments.net/reader037/viewer/2022102819/5681432e550346895daf9f47/html5/thumbnails/6.jpg)
Learning Rules
By learning we mean a procedure for modifying the weights and biases of a network
The training algorithm falls into three broad categories Supervised Learning
Here we need a set of example for the NN. {p1,t1}, {p2,t2},…, {pQ,tQ}
Where pq is the input and tq is the corresponding correct output.
Reinforcement Learning Similar to supervised but instead of the given correct output, only a
grade (score) is given. This is currently much less common than supervised.
Unsupervised Learning Here the weights and biases are modified in response to network
inputs only. Most of these algorithms perform clustering operation.
![Page 7: Artificial Neural Networks (1)](https://reader037.fdocuments.net/reader037/viewer/2022102819/5681432e550346895daf9f47/html5/thumbnails/7.jpg)
Typical Transfer Functions
![Page 8: Artificial Neural Networks (1)](https://reader037.fdocuments.net/reader037/viewer/2022102819/5681432e550346895daf9f47/html5/thumbnails/8.jpg)
Typical Transfer Functions
The log-sigmoid transfer function is commonly used in multilayer networks that are trained using the back-propagation algorithm, in part because this function is differentiable.
![Page 9: Artificial Neural Networks (1)](https://reader037.fdocuments.net/reader037/viewer/2022102819/5681432e550346895daf9f47/html5/thumbnails/9.jpg)
Example on Single-Input Neuron
The input to a single input neuron is 2.0, its weights is 2.3 and its bias is -3. What is the net input to the transfer function?
n=w p + b = (2.3) (2) + (-3) = 1.6
What is the neuron output, if it has the following transfer functions? Hard limit Linear Log-sigmoid
a= hardlim (1.6) =1.0
a= purelin (1.6) =1.6
a= logsig (1.6) = 1/ (1+e-
1.6)=0.8320
![Page 10: Artificial Neural Networks (1)](https://reader037.fdocuments.net/reader037/viewer/2022102819/5681432e550346895daf9f47/html5/thumbnails/10.jpg)
Multi-Input Neuron
n= w1,1 p1 + w1,2 p2 + w1,3 p3 + … +w1,R pR + b1
n=W p + b
First Index for the Neuron
Second Index for the Input
![Page 11: Artificial Neural Networks (1)](https://reader037.fdocuments.net/reader037/viewer/2022102819/5681432e550346895daf9f47/html5/thumbnails/11.jpg)
Multi-Input Neuron
The number of input is set by the external specifications of the problem.
If you want to design a neural network that is to predict kite-flying conditions and the inputs are air temperature, wind velocity, and humidity, then there would be three inputs to the network (R=3).
![Page 12: Artificial Neural Networks (1)](https://reader037.fdocuments.net/reader037/viewer/2022102819/5681432e550346895daf9f47/html5/thumbnails/12.jpg)
Example on Multi-Input Neuron
Given a two-input neuron with the following parameters: b=1.2, W=[3 2] and p=[-5 6]T, calculate the neuron output for the following transfer functions: A symmetrical hard limit transfer fn. A saturating linear transfer fn. A hyperbolic tangent sigmoid transfer fn.
n= W p + b = [ 3 2 ] * [ -5 6]T + (1.2)= -1.8 a= hardlims (-1.8)=-1 a= satlin(-1.8)=0 a= transig(-1.8)=0.9468
![Page 13: Artificial Neural Networks (1)](https://reader037.fdocuments.net/reader037/viewer/2022102819/5681432e550346895daf9f47/html5/thumbnails/13.jpg)
S Neurons with R Inputs each
![Page 14: Artificial Neural Networks (1)](https://reader037.fdocuments.net/reader037/viewer/2022102819/5681432e550346895daf9f47/html5/thumbnails/14.jpg)
S Neurons with R Inputs each
![Page 15: Artificial Neural Networks (1)](https://reader037.fdocuments.net/reader037/viewer/2022102819/5681432e550346895daf9f47/html5/thumbnails/15.jpg)
Example on Single-Neuron Perceptron
Ai= hardlim (ni) = hardlim(iwT p + bi) Given w1,1=1, w1,2=1, b= - 1
The decision boundary (line at which n=0) n=1wT p + b1=w1,1p1+w1,2p2+b =p1+p2-1 = 0
On one side of this boundary, the output will be 0; on the line and on the other side the output is 1
Test with any point to know the direction The boundary is always orthogonal to 1w
![Page 16: Artificial Neural Networks (1)](https://reader037.fdocuments.net/reader037/viewer/2022102819/5681432e550346895daf9f47/html5/thumbnails/16.jpg)
Adjusting the Bias
If the decision boundary line is given then we can adjust the bias according to the following equation:
1wT p + b1= 0
![Page 17: Artificial Neural Networks (1)](https://reader037.fdocuments.net/reader037/viewer/2022102819/5681432e550346895daf9f47/html5/thumbnails/17.jpg)
Design Problem
Lets design an AND gate using NN
1) Draw the input spaceBlack dots --> output=1
White dots --> output=0
![Page 18: Artificial Neural Networks (1)](https://reader037.fdocuments.net/reader037/viewer/2022102819/5681432e550346895daf9f47/html5/thumbnails/18.jpg)
Design Problem cont.
2. Select a decision boundarythere are infinite number of lines
that separates the black dots from
the white dots.
BUT it’s reasonable to choose a line halfway
3. Choose weight vector that is orthogonal
to the decision boundary
the weight vector can be of any length
so there are infinite possibilities
One choice is 1wT=[2 2]
![Page 19: Artificial Neural Networks (1)](https://reader037.fdocuments.net/reader037/viewer/2022102819/5681432e550346895daf9f47/html5/thumbnails/19.jpg)
Design Problem cont.
4. Find the biasPick any point on the decision boundary
line and substitute in
1wT p + b1= 0
For example, p=[1.5 0]T
1wT p + b1= [2 2]*[1.5 0]T+b = 3 + b =0
b= -3
5. Test the network
![Page 20: Artificial Neural Networks (1)](https://reader037.fdocuments.net/reader037/viewer/2022102819/5681432e550346895daf9f47/html5/thumbnails/20.jpg)
Multiple-Neuron Perceptron
There will be one decision boundary for EACH neuron.
The decision boundaries will be defined by
iwT p + bi= 0
![Page 21: Artificial Neural Networks (1)](https://reader037.fdocuments.net/reader037/viewer/2022102819/5681432e550346895daf9f47/html5/thumbnails/21.jpg)
Example on Supervision Learning
The problem has two inputs and one output
Therefore two inputs and one neuron
![Page 22: Artificial Neural Networks (1)](https://reader037.fdocuments.net/reader037/viewer/2022102819/5681432e550346895daf9f47/html5/thumbnails/22.jpg)
Example on Supervision Learning cont.
Lets start without any bias, so only two parameters w1,1 and w1,2
Removing the bias, then decision boundary passes through the origin
![Page 23: Artificial Neural Networks (1)](https://reader037.fdocuments.net/reader037/viewer/2022102819/5681432e550346895daf9f47/html5/thumbnails/23.jpg)
Example on Supervision Learning cont.
We want a learning rule that will find a weight vector that points in one of these directions (length of vector is not important)
So lets start with random weights
1wT=[1.0 -0.8]
Then present it to the network with p1. a=hardlim( [1.0 -0.8]*[1 2]T
a=hardlim( -0.6 ) = 0 The NN has made a mistake!
![Page 24: Artificial Neural Networks (1)](https://reader037.fdocuments.net/reader037/viewer/2022102819/5681432e550346895daf9f47/html5/thumbnails/24.jpg)
Example on Supervision Learning cont.
Why did it make a mistake?
If we adjust the weights but adding p1 to it
this would make 1w point more in the direction
of p1 so that it would hopefully not
be classified falsely into the wrong zone.
21
3 1w
SO If ( t=1 and a=0 ), then 1wnew = 1wold + p
![Page 25: Artificial Neural Networks (1)](https://reader037.fdocuments.net/reader037/viewer/2022102819/5681432e550346895daf9f47/html5/thumbnails/25.jpg)
Example on Supervision Learning cont.
Test the next input p2 with new weights a=hardlim( [2.0 1.2]*[-1 2]T
a=hardlim( 0.4 ) = 1
Again a mistake.
This time we want w to move away from p2
21
3 1w
1wnew = 1wold + p = [ 1.0 -0.8 ]T + [ 1 2 ]T= [ 2.0 1.2 ]T
SO If ( t=0 and a=1 ), then 1wnew = 1wold - p
![Page 26: Artificial Neural Networks (1)](https://reader037.fdocuments.net/reader037/viewer/2022102819/5681432e550346895daf9f47/html5/thumbnails/26.jpg)
Example on Supervision Learning cont.
Test the next input p3 with new weights a=hardlim( [3.0 -0.8 ]*[0 -1]T
a=hardlim( 0.8 ) = 1
Again a mistake.
We want w to move away from p3
21
3 1w
1wnew = 1wold + p = [2.0 1.2]T - [- 1 2 ]T= [ 3.0 -0.8 ]T
SO If ( t=0 and a=1 ), then 1wnew = 1wold - p
![Page 27: Artificial Neural Networks (1)](https://reader037.fdocuments.net/reader037/viewer/2022102819/5681432e550346895daf9f47/html5/thumbnails/27.jpg)
Example on Supervision Learning cont.
Test the next input p1 again with new weights a=hardlim( [ 3.0 0.2 ]*[1 2]T
a=hardlim( 3.4 ) = 1
Now correct.
Repeat for all other inputs
21
3 1w
1wnew = 1wold + p = [3.0 -0.8 ]T - [0 -1 ]T= [ 3.0 0.2 ]T
SO If ( t=a1) then wnew = 1wold
![Page 28: Artificial Neural Networks (1)](https://reader037.fdocuments.net/reader037/viewer/2022102819/5681432e550346895daf9f47/html5/thumbnails/28.jpg)
Multilayer Network
Two Hidden Layers One Output Layer
![Page 29: Artificial Neural Networks (1)](https://reader037.fdocuments.net/reader037/viewer/2022102819/5681432e550346895daf9f47/html5/thumbnails/29.jpg)
Recurrent Network
It is a network with feedback.
Some of its outputs is connected to its inputs
a(1)=satlins ( W a(0) + b )
a(2)=satlins ( W a(1) + b ), …