1 Computation in neural networks M. Meeter. 2 Perceptron learning problem Input Patterns Desired...
-
Upload
chance-charter -
Category
Documents
-
view
239 -
download
3
Transcript of 1 Computation in neural networks M. Meeter. 2 Perceptron learning problem Input Patterns Desired...
![Page 1: 1 Computation in neural networks M. Meeter. 2 Perceptron learning problem Input Patterns Desired output [+1, +1, -1, -1] [+1, -1, +1] [-1, -1, +1, +1]](https://reader036.fdocuments.net/reader036/viewer/2022081420/551b1e8f550346cf5a8b5706/html5/thumbnails/1.jpg)
1
Computation in neural networks
M. Meeter
![Page 2: 1 Computation in neural networks M. Meeter. 2 Perceptron learning problem Input Patterns Desired output [+1, +1, -1, -1] [+1, -1, +1] [-1, -1, +1, +1]](https://reader036.fdocuments.net/reader036/viewer/2022081420/551b1e8f550346cf5a8b5706/html5/thumbnails/2.jpg)
2
Perceptron learning problem
Input Patterns Desired output
[+1, +1, -1, -1] [+1, -1, +1]
[-1, -1, +1, +1] [+1, +1, -1]
[-1, -1, -1, -1]
[-1, -1, +1, -1] [-1, -1, -1]
[-1, +1, +1, -1] [-1, +1, +1]
[+1, -1, +1, -1]
Calculating a function
![Page 3: 1 Computation in neural networks M. Meeter. 2 Perceptron learning problem Input Patterns Desired output [+1, +1, -1, -1] [+1, -1, +1] [-1, -1, +1, +1]](https://reader036.fdocuments.net/reader036/viewer/2022081420/551b1e8f550346cf5a8b5706/html5/thumbnails/3.jpg)
3
Types of networks & functions
Attractor
Feedfwrd Hebbian
•associative (Hebbian)
•competitive
Feedfwrd error corr.
•perceptron
•backprop
completion, autoass. memory
•association, assoc. memory
•clustering
•categorization, generalization
•nonlinear, same
![Page 4: 1 Computation in neural networks M. Meeter. 2 Perceptron learning problem Input Patterns Desired output [+1, +1, -1, -1] [+1, -1, +1] [-1, -1, +1, +1]](https://reader036.fdocuments.net/reader036/viewer/2022081420/551b1e8f550346cf5a8b5706/html5/thumbnails/4.jpg)
4
Types of networks
Attractor
Feedfwrd Hebbian
•associative (Hebbian)
•competitive
Feedfwrd error corr.
•perceptron
•backprop
completion, autoass. memory
•association, assoc. memory
•clustering
•categorization, generalization
•nonlinear, same
![Page 5: 1 Computation in neural networks M. Meeter. 2 Perceptron learning problem Input Patterns Desired output [+1, +1, -1, -1] [+1, -1, +1] [-1, -1, +1, +1]](https://reader036.fdocuments.net/reader036/viewer/2022081420/551b1e8f550346cf5a8b5706/html5/thumbnails/5.jpg)
5
Classification
1~x
A 1~x
2~x
![Page 6: 1 Computation in neural networks M. Meeter. 2 Perceptron learning problem Input Patterns Desired output [+1, +1, -1, -1] [+1, -1, +1] [-1, -1, +1, +1]](https://reader036.fdocuments.net/reader036/viewer/2022081420/551b1e8f550346cf5a8b5706/html5/thumbnails/6.jpg)
6
Generalization
76
128
?
![Page 7: 1 Computation in neural networks M. Meeter. 2 Perceptron learning problem Input Patterns Desired output [+1, +1, -1, -1] [+1, -1, +1] [-1, -1, +1, +1]](https://reader036.fdocuments.net/reader036/viewer/2022081420/551b1e8f550346cf5a8b5706/html5/thumbnails/7.jpg)
7
Univariate Linear Regression
prediction of values
Y
X
)(
ˆ
ˆ
2
eKSMin
baxy
yye
Regression = generalization
![Page 8: 1 Computation in neural networks M. Meeter. 2 Perceptron learning problem Input Patterns Desired output [+1, +1, -1, -1] [+1, -1, +1] [-1, -1, +1, +1]](https://reader036.fdocuments.net/reader036/viewer/2022081420/551b1e8f550346cf5a8b5706/html5/thumbnails/8.jpg)
8
Clustering
1~x
2~x
![Page 9: 1 Computation in neural networks M. Meeter. 2 Perceptron learning problem Input Patterns Desired output [+1, +1, -1, -1] [+1, -1, +1] [-1, -1, +1, +1]](https://reader036.fdocuments.net/reader036/viewer/2022081420/551b1e8f550346cf5a8b5706/html5/thumbnails/9.jpg)
9
Types of networks
Attractor
Feedfwrd Hebbian
•associative (Hebbian)
•competitive
Feedfwrd error corr.
•perceptron
•backprop
completion, autoass. memory
•association, assoc. memory
•clustering
•categorization, generalization
•nonlinear, same
![Page 10: 1 Computation in neural networks M. Meeter. 2 Perceptron learning problem Input Patterns Desired output [+1, +1, -1, -1] [+1, -1, +1] [-1, -1, +1, +1]](https://reader036.fdocuments.net/reader036/viewer/2022081420/551b1e8f550346cf5a8b5706/html5/thumbnails/10.jpg)
10
Perceptron learning problem
Prototypical Input Patterns Desired output
[+1, +1, -1, -1] [+1, -1, +1]
[-1, -1, +1, +1] [+1, +1, -1]
[-1, -1, -1, -1]
[-1, -1, +1, -1] [-1, -1, -1]
[-1, +1, +1, -1] [-1, +1, +1]
[+1, -1, +1, -1]
Classification - discrete
![Page 11: 1 Computation in neural networks M. Meeter. 2 Perceptron learning problem Input Patterns Desired output [+1, +1, -1, -1] [+1, -1, +1] [-1, -1, +1, +1]](https://reader036.fdocuments.net/reader036/viewer/2022081420/551b1e8f550346cf5a8b5706/html5/thumbnails/11.jpg)
11
Perceptron learning problem
Prototypical Input Patterns Desired output
[+1, +1, -1, -1] [+1, -1, +1]
[-1, -1, +1, +1] [+1, +1, -1]
[-1, -1, -1, -1]
[-1, -1, +1, -1] [-1, -1, -1]
[-1, +1, +1, -1] [-1, +1, +1]
[+1, -1, +1, -1]
Classification - discrete
![Page 12: 1 Computation in neural networks M. Meeter. 2 Perceptron learning problem Input Patterns Desired output [+1, +1, -1, -1] [+1, -1, +1] [-1, -1, +1, +1]](https://reader036.fdocuments.net/reader036/viewer/2022081420/551b1e8f550346cf5a8b5706/html5/thumbnails/12.jpg)
12
Xi
X1
X2
Xn
wji
0 0
0 1)
*
vif
vifv
wxvi
jii
1~x
threshold
y
Classification in Perceptron
![Page 13: 1 Computation in neural networks M. Meeter. 2 Perceptron learning problem Input Patterns Desired output [+1, +1, -1, -1] [+1, -1, +1] [-1, -1, +1, +1]](https://reader036.fdocuments.net/reader036/viewer/2022081420/551b1e8f550346cf5a8b5706/html5/thumbnails/13.jpg)
13
Effe tussendoor…
Bij perceptron etc.: net input knoop>0 dan activatie 0
Niet altijd gewenst: daarom heeft knoop in continue vormen perceptron / backprop een ‘bias’, een activatie die altijd bij input opgeteld wordt
Effect: verschuiven threshold
![Page 14: 1 Computation in neural networks M. Meeter. 2 Perceptron learning problem Input Patterns Desired output [+1, +1, -1, -1] [+1, -1, +1] [-1, -1, +1, +1]](https://reader036.fdocuments.net/reader036/viewer/2022081420/551b1e8f550346cf5a8b5706/html5/thumbnails/14.jpg)
14
Classification in 2 dimensions
1~x
2~x
1~xThreshold Input=
ThresholdInput= mixture
+
-
![Page 15: 1 Computation in neural networks M. Meeter. 2 Perceptron learning problem Input Patterns Desired output [+1, +1, -1, -1] [+1, -1, +1] [-1, -1, +1, +1]](https://reader036.fdocuments.net/reader036/viewer/2022081420/551b1e8f550346cf5a8b5706/html5/thumbnails/15.jpg)
15
Discriminant Analysis
1~x
2~x
11gx 21gx
12gx
22gx
Produces exact same result
Find center of two categories, draw line in between, then one diagonal in middle = discrimination line
![Page 16: 1 Computation in neural networks M. Meeter. 2 Perceptron learning problem Input Patterns Desired output [+1, +1, -1, -1] [+1, -1, +1] [-1, -1, +1, +1]](https://reader036.fdocuments.net/reader036/viewer/2022081420/551b1e8f550346cf5a8b5706/html5/thumbnails/16.jpg)
16
Univariate Linear Regression
prediction of values
Y
X
)(
ˆ
ˆ
2
eKSMin
baxy
yye
Generalization = Regression
![Page 17: 1 Computation in neural networks M. Meeter. 2 Perceptron learning problem Input Patterns Desired output [+1, +1, -1, -1] [+1, -1, +1] [-1, -1, +1, +1]](https://reader036.fdocuments.net/reader036/viewer/2022081420/551b1e8f550346cf5a8b5706/html5/thumbnails/17.jpg)
17
Xi
Activation function
(·)
X1
X2
Xn
y
Change weights with rule, minimizing e2
j
wji
v = xi*wji
(v) = av + b
Bias
y
Perceptron with linear activation rule
![Page 18: 1 Computation in neural networks M. Meeter. 2 Perceptron learning problem Input Patterns Desired output [+1, +1, -1, -1] [+1, -1, +1] [-1, -1, +1, +1]](https://reader036.fdocuments.net/reader036/viewer/2022081420/551b1e8f550346cf5a8b5706/html5/thumbnails/18.jpg)
18
Multivariate = multiple independent variables X
=multiple inputsXi
X1
X2
Xn
1 y1
2 y2
X
Y1
Y2
y
y
Multivariate Multiple Linear Regression
Multiple = multiple dependent variables Y
=multiple outputs
![Page 19: 1 Computation in neural networks M. Meeter. 2 Perceptron learning problem Input Patterns Desired output [+1, +1, -1, -1] [+1, -1, +1] [-1, -1, +1, +1]](https://reader036.fdocuments.net/reader036/viewer/2022081420/551b1e8f550346cf5a8b5706/html5/thumbnails/19.jpg)
19
Linear vs. nonlinear regression
linear
x
ynonlinear
x
y
Here: quadraticGeneral: wrinkle-fitting
![Page 20: 1 Computation in neural networks M. Meeter. 2 Perceptron learning problem Input Patterns Desired output [+1, +1, -1, -1] [+1, -1, +1] [-1, -1, +1, +1]](https://reader036.fdocuments.net/reader036/viewer/2022081420/551b1e8f550346cf5a8b5706/html5/thumbnails/20.jpg)
20
y1
y2
X
X
X= [x1, x2, .., xi, .., xn]
*wxvi
jii
avev
1
1)(1y
2y
Multi-Layer Perceptron
Fit any function:“Universal approximators”
![Page 21: 1 Computation in neural networks M. Meeter. 2 Perceptron learning problem Input Patterns Desired output [+1, +1, -1, -1] [+1, -1, +1] [-1, -1, +1, +1]](https://reader036.fdocuments.net/reader036/viewer/2022081420/551b1e8f550346cf5a8b5706/html5/thumbnails/21.jpg)
21
x
y
Too simple model
Bad
Too complex model
x
y
Extremely bad
Overfitting
![Page 22: 1 Computation in neural networks M. Meeter. 2 Perceptron learning problem Input Patterns Desired output [+1, +1, -1, -1] [+1, -1, +1] [-1, -1, +1, +1]](https://reader036.fdocuments.net/reader036/viewer/2022081420/551b1e8f550346cf5a8b5706/html5/thumbnails/22.jpg)
22
Clustering
1~x
2~x
Competitive learning:next weekART
![Page 23: 1 Computation in neural networks M. Meeter. 2 Perceptron learning problem Input Patterns Desired output [+1, +1, -1, -1] [+1, -1, +1] [-1, -1, +1, +1]](https://reader036.fdocuments.net/reader036/viewer/2022081420/551b1e8f550346cf5a8b5706/html5/thumbnails/23.jpg)
23
Conclusions
Neural networks similar to statistical analyses Perceptron -> categorization / generalization Backprop -> same but nonlinear Competitive l. -> clustering
But… Whole data set vs. one pattern at a time
![Page 24: 1 Computation in neural networks M. Meeter. 2 Perceptron learning problem Input Patterns Desired output [+1, +1, -1, -1] [+1, -1, +1] [-1, -1, +1, +1]](https://reader036.fdocuments.net/reader036/viewer/2022081420/551b1e8f550346cf5a8b5706/html5/thumbnails/24.jpg)
24
Feature reduction with PCA
1~x
2~x
11gx 21gx
12gx
22gx
![Page 25: 1 Computation in neural networks M. Meeter. 2 Perceptron learning problem Input Patterns Desired output [+1, +1, -1, -1] [+1, -1, +1] [-1, -1, +1, +1]](https://reader036.fdocuments.net/reader036/viewer/2022081420/551b1e8f550346cf5a8b5706/html5/thumbnails/25.jpg)
25
Feature extraction with PCA
1y
1y
1y
1y
?
?
Unsupervised Learning
Hebbian Learning