Neural Network Concepts and Paradigms · Computational Intelligence in Complex Decision Systems G....
Transcript of Neural Network Concepts and Paradigms · Computational Intelligence in Complex Decision Systems G....
Computational Intelligence in Complex Decision Systems G. Oltean
Neural Network Concepts
and Paradigms
Computational Intelligence in Complex Decision Systems G. Oltean
Biological NeuronNeurons: nerve cells; consist of dendrites, body and an axon;
signals flow through synapses.
Axon(output)
dendrites => body => axon
Axons connects to dendrites via synapses Synapses
vary in strength may be excitatory or inhibitory (+ or -)
an electrically excitable cell that processes and transmits
information through electrical and chemical signals.
Computational Intelligence in Complex Decision Systems G. Oltean
Human Brain
on average the human brain has
• on average 86 billions neurons
• ~1000 main modules /~500 neural network each /~100 000 neurons
Properties of the brain:
It can learn, reorganize itself from experience
It adapts to the environment
It is robust and fault tolerant
Computational Intelligence in Complex Decision Systems G. Oltean
Artificial Neural Networks
Some differences between biological neurons and
artificial neurons (processing elements - PE):
* Signs of weights (+ or -)
* Signals are ac in biological neurons, dc in PEs
* Many types of neurons in a system; usually only
a few at most in neural networks
* Basic cycle time for PC (<1 ns) faster than
brain (10-100ms) {as far as we know!}
An analysis paradigm very roughly modeled after themassively parallel structure of the brain.
Simulates a highly interconnected, parallel computationalstructure with numerous relatively simple individualprocessing elements (PE ).
Computational Intelligence in Complex Decision Systems G. Oltean
What ANNs are and why they are useful
In the simplest terms, ANNs map input vectors to output vectors
ANNs consist of processing elements (PEs) or
neurons and weighted connections
• Operations performed by ANNs can include:* Fitting (approximation) and modeling* Pattern recognition (Classification)* Prediction* Pattern matching and completion* Noise removal* Optimization* Simulation* Control
Computational Intelligence in Complex Decision Systems G. Oltean
What NNs are and why they are useful
• Connection weights store the information
• Weight values usually determined by a learning procedure
• Each PE acts independently of all others
• Each PE relies only on local information
• Connection pattern provides redundancy and facilitates fault tolerance
• ANNs are able to “learn” arbitrary non-linear mappings
Computational Intelligence in Complex Decision Systems G. Oltean
7 / 20
Housing Price
Prediction
https://www.coursera.org/learn/neural-networks-deep-learning/lecture/eAE2G/what-is-a-neural-network
Computational Intelligence in Complex Decision Systems G. Oltean
8 / 20
https://www.coursera.org/learn/neural-networks-deep-learning/lecture/2c38r/supervised-learning-with-neural-networks
Computational Intelligence in Complex Decision Systems G. Oltean
9 / 20
https://www.coursera.org/learn/neural-networks-deep-learning/lecture/2c38r/supervised-learning-with-neural-networks
Computational Intelligence in Complex Decision Systems G. Oltean
10 / 20
https://www.coursera.org/learn/neural-networks-deep-learning/lecture/2c38r/supervised-learning-with-neural-networks
People are just really good at interpreting unstructured data.
It has been much harder for computers to make sense of unstructured data compared to structured data.
Thanks to deep learning, thanks to neural networks, computers are now much better at interpreting unstructured data
Computational Intelligence in Complex Decision Systems G. Oltean
11 / 20
Computational Intelligence in Complex Decision Systems G. Oltean
Feedforward Neural Network
Input layer Hidden layer Output layer
Computational Intelligence in Complex Decision Systems G. Oltean
Principal elements required to specify ANNs
Topology - layers and interconnections
Learning - how network is configured to store information
Recall - How stored information is retrieved
Computational Intelligence in Complex Decision Systems G. Oltean
Neuron (PE) Model
- summing component
f - activation function
𝑠 = 𝑤1𝑥1 + 𝑤2𝑥2 + …+ 𝑤𝑁𝑥𝑁 + 𝑏
If N = 1, the weighted sum is in fact the simple linear regression
𝑠 = 𝑤1𝑥1 + 𝑏
If N > 1, the weighted sum is in fact the multiple linear regression
b is the intercept
Computational Intelligence in Complex Decision Systems G. Oltean
Neuron (PE)
Model
TNxxxx ,...,, 21
Nwwww ,...,, 21
Input vector
Weight vector
Bias
N
i
ii bxwfysfy1
)(
bwxsbxwsN
i
ii 1
- summing component
f - activation function
b
multiple linear regression
Computational Intelligence in Complex Decision Systems G. Oltean
Problem
5.2];5.025.0[
;4
2;2
bw
xN
)(sf
s = ?; y = ?
Implement in a programing language the above neuron
Computational Intelligence in Complex Decision Systems G. Oltean
Neuron Model as Logistic Regression
TNxxxx ,...,, 21Input vector
Binary classification
1 – positive (in the class)
0 – negative (not in the class)
features vector (of the objects)
Output 𝑦 ∈ 0, 1
We want our model to predict the chance for the object
described by a set of features x to be in the class
𝑦 ∈ [0, 1] 0 ≤ 𝑦 ≤ 1
Computational Intelligence in Complex Decision Systems G. Oltean
Neuron Model as Logistic Regression
TNxxxx ,...,, 21
Binary classification
𝑦 ∈ [0, 1] 0 ≤ 𝑦 ≤ 1
Activation function f
Sigmoidal (logistic function)
𝑦 =1
1+e−𝑠
Computational Intelligence in Complex Decision Systems G. Oltean
19 / 20
Phyton 3.5 (Spyder) implementation using NumPy module
Computational Intelligence in Complex Decision Systems G. Oltean
20 / 20
Phyton 3.5 (Spyder) results
x=24
w = [0.25, -0.5]b = 2.5
s= 1.0y = 0.7615941559557649
x=24-0.3
w = [0.25, -0.5, 1]b = 0.5
s= -1.3y = -0.8617231593133063
x=24-0.8
w = [0.25, -0.5, 1]b = -0.99
s=-3.29y = -0.9972281483227239
Computational Intelligence in Complex Decision Systems G. Oltean
Activation(Transfer) Functions
hardlims
0,1
0,1)(
s
ssy
)(sf
)(sf
Computational Intelligence in Complex Decision Systems G. Oltean
Activation(Transfer) Functions
)(sf
)(sf
Computational Intelligence in Complex Decision Systems G. Oltean
Simulink Transfer Functions
Computational Intelligence in Complex Decision Systems G. Oltean
ANN with 1 layer 2 inputs, 3 outputs
2
1
x
xx
3231
2221
1211
ww
ww
ww
w
3
2
1
b
b
b
b
3
2
1
s
s
s
s
3
2
1
o
o
o
o
bwxs )(sfo
3
2
1
2
1
3231
2221
1211
3
2
1
b
b
b
x
x
ww
ww
ww
s
s
s
Computational Intelligence in Complex Decision Systems G. Oltean
Exercise
3
5.0
5.0
5.075.0
11.0
15.0
bw
0
10
4
2
4
1321 xxx
sesf
1
1)(
Compute the output vector o, for each input vector xi
Implement in a programing language the above example
Computational Intelligence in Complex Decision Systems G. Oltean
26 / 20
Phyton 3.5 (Spyder) implementation
Computational Intelligence in Complex Decision Systems G. Oltean
27 / 20
The one-layer user-defined function is written in a separate
file (ANNfunc.py)
Computational Intelligence in Complex Decision Systems G. Oltean
28 / 20
The input is:[[-1. ][-2. ][ 0.33]]
The weights matrix is:[[-0.5 -0.15 -0.1 ][ 0.23 0.11 -0.5 ][-0.5 1. -0.1 ][ 0.045 0.15 -0.05 ]]
The bias is:[[ 0.2 ][-0.5 ][-0.03][ 0.36]]
The activation function is: tanh
The output is:[[ 0.747383 ][-0.80582272][-0.91590514][-0.0015 ]]
Phyton 3.5 (Spyder) results
Computational Intelligence in Complex Decision Systems G. Oltean
It is common for different layers to have different numbers of neurons.
Multiple-layer networks are quite powerful.
Feedforward Neural Network
An ANN of two layers, where the first layer is sigmoid and the second layer is linear, can be trained to approximate any function (with a finite number of discontinuities) arbitrarily well - universal aproximator
Computational Intelligence in Complex Decision Systems G. Oltean
Two layers: one hidden layer and one output layer
N inputs; H neurons in the hidden layer;
K outputs (neurons in the output layer)
hidden
layer
output
layer
output input
1x
hW
2x
Nx
1
hb1
hs1 hy1
1
1
hf
1
1
1
1o
oW
hs2
h
Hs
hf
hf
of
of
of
hb2
Hb
hy2
h
Hy
os1
os2
o
Ks
2o
Ko
ob1
ob2
o
Kb
∑
∑
∑
∑
∑
∑
Computational Intelligence in Complex Decision Systems G. Oltean
hidden layer:
h
HN
h
H
h
H
h
N
hh
h
N
hh
h
www
www
www
w
21
22221
11211
a
H
a
a
a
b
b
b
b2
1
hidden
layer
output
layer
output input
1x
hW
2x
Nx
1
hb1
hs1 hy1
1
1
hf
1
1
1
1o
oW
hs2
h
Hs
hf
hf
of
of
of
hb2
Hb
hy2
h
Hy
os1
os2
o
Ks
2o
Ko
ob1
ob2
o
Kb
∑
∑
∑
∑
∑
∑
Computational Intelligence in Complex Decision Systems G. Oltean
output layer:
o
KH
o
K
o
K
o
H
oo
o
H
oo
o
www
www
www
w
21
22221
11211
o
K
o
o
o
b
b
b
b2
1
hidden
layer
output
layer
output input
1x
hW
2x
Nx
1
hb1
hs1 hy1
1
1
hf
1
1
1
1o
oW
hs2
h
Hs
hf
hf
of
of
of
hb2
Hb
hy2
h
Hy
os1
os2
o
Ks
2o
Ko
ob1
ob2
o
Kb
∑
∑
∑
∑
∑
∑
Computational Intelligence in Complex Decision Systems G. Oltean
hhhh bxwfy Th
H
hhh yyyy 21
ohoo bywfO
ohhhoo bbxwfwfO
Transfer function:
hidden
layer
output
layer
output input
1x
hW
2x
Nx
1
hb1
hs1 hy1
1
1
hf
1
1
1
1o
oW
hs2
h
Hs
hf
hf
of
of
of
hb2
Hb
hy2
h
Hy
os1
os2
o
Ks
2o
Ko
ob1
ob2
o
Kb
∑
∑
∑
∑
∑
∑
Computational Intelligence in Complex Decision Systems G. Oltean
Exporting an ANN trained in Matlabto simulate it in another environment (e.g. Python)
In Matlab, training a ANN generally involves a normalization step that is
applied to both the input vectors and the target vectors in the data set.
The network output always falls into a normalized range, [-1, 1].
The network output can then be reverse transformed back into the units
of the original target data when the network is put to use in the field
(simulated in Matlab).
For correct export of the ANN, remove all preprocessing and
postprocessing functions that MATLAB executes on inputs and outputs,
(or implement those function in the new environment)
For a two-layer ANN, in the script (function) used to create and train the
ANN, write:
net.inputs{1}.processFcns = {}; % for inputs
net.outputs{2}.processFcns = {}; % for targets
Default setting is: net.inputs{1}.processFcns = {'removeconstantrows','mapminmax'};
net.outputs{2}.processFcns = {'removeconstantrows','mapminmax’};
Computational Intelligence in Complex Decision Systems G. Oltean
Exporting an ANN trained in Matlabto simulate it in another environment (e.g. Python) – cont.
The parameters of the trained ANN (e.g. for a two-layer ANN)
wh=cell2mat(net.IW) %weights of the hidden layer
wo=cell2mat(net.LW) %weights of the output layer
bh=cell2mat(net.b(1)) % bias of the hidden layer
bo=cell2mat(net.b(2)) % bias of the output layer
afunh=net.layers{1}.transferFcn % activation function in the
hidden layer
afunh=net.layers{2}.transferFcn % activation function in the
output layer
In Matlab:
Observation: “net” is the name of the ANN