Reading Assignmentsilab.usc.edu/classes/2002cs564/lecture_notes/17-Examples-Review.p… · FOn....
Transcript of Reading Assignmentsilab.usc.edu/classes/2002cs564/lecture_notes/17-Examples-Review.p… · FOn....
![Page 1: Reading Assignmentsilab.usc.edu/classes/2002cs564/lecture_notes/17-Examples-Review.p… · FOn. Laurent Itti: CS564 - Brain Theory and Artificial Intelligence. Saccades 2 7 Neuron](https://reader033.fdocuments.net/reader033/viewer/2022060710/607699a78e6a3057b8035a97/html5/thumbnails/1.jpg)
1Laurent Itti: CS564 - Brain Theory and Artificial Intelligence. Saccades 2
L. Itti: CS564 - Brain Theory and Artificial IntelligenceUniversity of Southern California
Lecture 17. Examples and Review
Reading Assignments:None
![Page 2: Reading Assignmentsilab.usc.edu/classes/2002cs564/lecture_notes/17-Examples-Review.p… · FOn. Laurent Itti: CS564 - Brain Theory and Artificial Intelligence. Saccades 2 7 Neuron](https://reader033.fdocuments.net/reader033/viewer/2022060710/607699a78e6a3057b8035a97/html5/thumbnails/2.jpg)
2Laurent Itti: CS564 - Brain Theory and Artificial Intelligence. Saccades 2
Syllabus Overview
Introduction
Overview
Charting the brain
The Brain as a Network of Neuronsx (t)1
x (t)n
x (t)2
y(t+1)
w1
2
n
w
w
axonθθθθ
![Page 3: Reading Assignmentsilab.usc.edu/classes/2002cs564/lecture_notes/17-Examples-Review.p… · FOn. Laurent Itti: CS564 - Brain Theory and Artificial Intelligence. Saccades 2 7 Neuron](https://reader033.fdocuments.net/reader033/viewer/2022060710/607699a78e6a3057b8035a97/html5/thumbnails/3.jpg)
3Laurent Itti: CS564 - Brain Theory and Artificial Intelligence. Saccades 2
Syllabus Overview
Introduction (cont.)
Experimental techniques
Introduction to Vision
Schemas
![Page 4: Reading Assignmentsilab.usc.edu/classes/2002cs564/lecture_notes/17-Examples-Review.p… · FOn. Laurent Itti: CS564 - Brain Theory and Artificial Intelligence. Saccades 2 7 Neuron](https://reader033.fdocuments.net/reader033/viewer/2022060710/607699a78e6a3057b8035a97/html5/thumbnails/4.jpg)
4Laurent Itti: CS564 - Brain Theory and Artificial Intelligence. Saccades 2
Syllabus Overview
Basic Neural Modeling & Adaptive Networks
Didday Model of Winner-Take-All
Hopfield networks
Adaptive networks: Hebbian learning;Perceptrons; landmark learning
E = - ½ Σ ij sisjwij + Σ i siθi
![Page 5: Reading Assignmentsilab.usc.edu/classes/2002cs564/lecture_notes/17-Examples-Review.p… · FOn. Laurent Itti: CS564 - Brain Theory and Artificial Intelligence. Saccades 2 7 Neuron](https://reader033.fdocuments.net/reader033/viewer/2022060710/607699a78e6a3057b8035a97/html5/thumbnails/5.jpg)
5Laurent Itti: CS564 - Brain Theory and Artificial Intelligence. Saccades 2
Syllabus Overview
Neural Modeling & Adaptive Networks (cont.)
Adaptive networks: gradient descentand backpropagation
Reinforcement learning
Competition and cooperation
Visual plasticity; self-organizing feature maps; Kohonen maps
↓ ↑ ↑ ↓ ↓ ↓↓ ↓ ↑ ↑ ↑ ↑↑ ↑ ↑ ↑ ↓ ↓
![Page 6: Reading Assignmentsilab.usc.edu/classes/2002cs564/lecture_notes/17-Examples-Review.p… · FOn. Laurent Itti: CS564 - Brain Theory and Artificial Intelligence. Saccades 2 7 Neuron](https://reader033.fdocuments.net/reader033/viewer/2022060710/607699a78e6a3057b8035a97/html5/thumbnails/6.jpg)
6Laurent Itti: CS564 - Brain Theory and Artificial Intelligence. Saccades 2
Syllabus Overview
Examples of Large-scale Neural Modeling
System concepts
Model of saccadic eye movements
Feedback and the spinal cord; mass-spring model of muscle
)(0
)()(
0010
)()(
)(1
tutxtx
txtx
tqm���
�
���
�+�
�
���
���
���
�=�
�
���
�=
���
��
PP
SC
vm
sm
ms
FOn
FEF
vm
sm
vs
CD
sm
vs
SNR
vsmsqv
TH
qv
LGN
VisCx
wta
Brainstem Saccade Generator
eye movement
RetinaVisInput
θ, θdelay
FEFvsFEFms
FEFvsFEFms
switc
hPPctr
θ, θdelay
FOn
![Page 7: Reading Assignmentsilab.usc.edu/classes/2002cs564/lecture_notes/17-Examples-Review.p… · FOn. Laurent Itti: CS564 - Brain Theory and Artificial Intelligence. Saccades 2 7 Neuron](https://reader033.fdocuments.net/reader033/viewer/2022060710/607699a78e6a3057b8035a97/html5/thumbnails/7.jpg)
7Laurent Itti: CS564 - Brain Theory and Artificial Intelligence. Saccades 2
Neuron Models
We have studied the following types of neurons:
Biological: very complex, activity depends on many factors (including presynaptic activity, topography of dendritic tree, ion channel densities, concentrations of neurotransmitters and other ions, etc). Not fully understood.
McCulloch & Pitts: binary output as thresholded weighted sum of inputs. Highly non-linear model.
Continuous extension (used in Hopfield & Backprop networks): continuous output as sigmoid’ed weighted sum of inputs.
Leaky integrator: adds explicit time evolution (RC circuit behavior, plus possible threshold and spiking mechanism).
![Page 8: Reading Assignmentsilab.usc.edu/classes/2002cs564/lecture_notes/17-Examples-Review.p… · FOn. Laurent Itti: CS564 - Brain Theory and Artificial Intelligence. Saccades 2 7 Neuron](https://reader033.fdocuments.net/reader033/viewer/2022060710/607699a78e6a3057b8035a97/html5/thumbnails/8.jpg)
8Laurent Itti: CS564 - Brain Theory and Artificial Intelligence. Saccades 2
Network Architectures
We have studied these major categories of network architectures:
Layered, feedforward networks with synchronous update and no loops
Hopfield networks with asynchronous update and symmetric weights
Self-organizing feature maps in which some local connectivity pattern yields interesting emergent global behavior
Arbitrary biologically-inspired networks with loops, e.g., the winner-take-all
![Page 9: Reading Assignmentsilab.usc.edu/classes/2002cs564/lecture_notes/17-Examples-Review.p… · FOn. Laurent Itti: CS564 - Brain Theory and Artificial Intelligence. Saccades 2 7 Neuron](https://reader033.fdocuments.net/reader033/viewer/2022060710/607699a78e6a3057b8035a97/html5/thumbnails/9.jpg)
9Laurent Itti: CS564 - Brain Theory and Artificial Intelligence. Saccades 2
System Architectures
We have started looking at system architecture issues:
The NSL simulation environment and modular, hierarchical development of complex neural models
Discussion of black-box vs. fully-engineered approaches
Notion of schemas as intermediaries between neural patterns of activity and mental events
The Dominey-Arbib model of saccadic eye movements
… and we will focus on studying more examples of complex, biologically-inspired models in the second part of the course.
![Page 10: Reading Assignmentsilab.usc.edu/classes/2002cs564/lecture_notes/17-Examples-Review.p… · FOn. Laurent Itti: CS564 - Brain Theory and Artificial Intelligence. Saccades 2 7 Neuron](https://reader033.fdocuments.net/reader033/viewer/2022060710/607699a78e6a3057b8035a97/html5/thumbnails/10.jpg)
10Laurent Itti: CS564 - Brain Theory and Artificial Intelligence. Saccades 2
Learning and Adaptation
Finally, we have studied the following adaptationschemes:
Hebbian learning (strenghtening by co-activation) and Pavlovian conditioning
Perceptron learning rule (strengthening based on comparison between actual output and desired output)
Backpropagation (to extend the perceptron learning rule to hidden units subject to the credit assignment problem)
Reinforcement learning (or learning through monitoring one’s own successes and failures, through a critic that may itself be adaptive)
![Page 11: Reading Assignmentsilab.usc.edu/classes/2002cs564/lecture_notes/17-Examples-Review.p… · FOn. Laurent Itti: CS564 - Brain Theory and Artificial Intelligence. Saccades 2 7 Neuron](https://reader033.fdocuments.net/reader033/viewer/2022060710/607699a78e6a3057b8035a97/html5/thumbnails/11.jpg)
11Laurent Itti: CS564 - Brain Theory and Artificial Intelligence. Saccades 2
Some Current Trends
In basic computational neuroscience, much currentwork goes into understanding the basic biophysics of computation. This typically involves much more detailed models and heavy simulations.
Issues of interest include:
- The computational role of specific dendritic tree structures- Spike timing and synchronization- Neuromodulation- Coupling and properties of small recurrent networks- Information-theoretic analysis of neurons and synapses- Biochemical bases of learning- … and many more.
![Page 12: Reading Assignmentsilab.usc.edu/classes/2002cs564/lecture_notes/17-Examples-Review.p… · FOn. Laurent Itti: CS564 - Brain Theory and Artificial Intelligence. Saccades 2 7 Neuron](https://reader033.fdocuments.net/reader033/viewer/2022060710/607699a78e6a3057b8035a97/html5/thumbnails/12.jpg)
12Laurent Itti: CS564 - Brain Theory and Artificial Intelligence. Saccades 2
The Cable Equation
See http://diwww.epfl.ch/~gerstner/SPNM/SPNM.htmlFor excellent additional material (some reproduced here).
Just a piece of passive dendrite can yield complicated differential equations which have been extensively studied by electronicians in the context of the study of coaxial cables (TV antenna cable):
![Page 13: Reading Assignmentsilab.usc.edu/classes/2002cs564/lecture_notes/17-Examples-Review.p… · FOn. Laurent Itti: CS564 - Brain Theory and Artificial Intelligence. Saccades 2 7 Neuron](https://reader033.fdocuments.net/reader033/viewer/2022060710/607699a78e6a3057b8035a97/html5/thumbnails/13.jpg)
13Laurent Itti: CS564 - Brain Theory and Artificial Intelligence. Saccades 2
The Hodgkin-Huxley Model
Adding active ion channels yields a fairly realistic description of axons, dendrites and neurons.
The Hodgkin-Huxley is an example of such fairly detailed model. It is an extension of the leaky integrator model, adding active ion channels. It is described by a set of coupled non-linear first-order differential equations. Simulating these equations yields fairly realistic time-dependent simulations.
![Page 14: Reading Assignmentsilab.usc.edu/classes/2002cs564/lecture_notes/17-Examples-Review.p… · FOn. Laurent Itti: CS564 - Brain Theory and Artificial Intelligence. Saccades 2 7 Neuron](https://reader033.fdocuments.net/reader033/viewer/2022060710/607699a78e6a3057b8035a97/html5/thumbnails/14.jpg)
14Laurent Itti: CS564 - Brain Theory and Artificial Intelligence. Saccades 2
The Hodgkin-Huxley Model
Example spike trains obtained…
![Page 15: Reading Assignmentsilab.usc.edu/classes/2002cs564/lecture_notes/17-Examples-Review.p… · FOn. Laurent Itti: CS564 - Brain Theory and Artificial Intelligence. Saccades 2 7 Neuron](https://reader033.fdocuments.net/reader033/viewer/2022060710/607699a78e6a3057b8035a97/html5/thumbnails/15.jpg)
15Laurent Itti: CS564 - Brain Theory and Artificial Intelligence. Saccades 2
Detailed Neural Modeling
A simulator, called “Neuron” has been developedat Yale to simulate the Hodgkin-Huxley equations,as well as other membranes/channels/etc.See http://www.neuron.yale.edu/
![Page 16: Reading Assignmentsilab.usc.edu/classes/2002cs564/lecture_notes/17-Examples-Review.p… · FOn. Laurent Itti: CS564 - Brain Theory and Artificial Intelligence. Saccades 2 7 Neuron](https://reader033.fdocuments.net/reader033/viewer/2022060710/607699a78e6a3057b8035a97/html5/thumbnails/16.jpg)
16Laurent Itti: CS564 - Brain Theory and Artificial Intelligence. Saccades 2
Detailed Neural Modeling
The Genesis model has been developed at Caltech tosimulate large, complex dendritic structures, usingcompartmental modeling.
See http://www.genesis-sim.org/GENESIS/
![Page 17: Reading Assignmentsilab.usc.edu/classes/2002cs564/lecture_notes/17-Examples-Review.p… · FOn. Laurent Itti: CS564 - Brain Theory and Artificial Intelligence. Saccades 2 7 Neuron](https://reader033.fdocuments.net/reader033/viewer/2022060710/607699a78e6a3057b8035a97/html5/thumbnails/17.jpg)
17Laurent Itti: CS564 - Brain Theory and Artificial Intelligence. Saccades 2
Applications of Neural Networks
See http://www.neusciences.com/Technologies/nn_intro.htm
![Page 18: Reading Assignmentsilab.usc.edu/classes/2002cs564/lecture_notes/17-Examples-Review.p… · FOn. Laurent Itti: CS564 - Brain Theory and Artificial Intelligence. Saccades 2 7 Neuron](https://reader033.fdocuments.net/reader033/viewer/2022060710/607699a78e6a3057b8035a97/html5/thumbnails/18.jpg)
18Laurent Itti: CS564 - Brain Theory and Artificial Intelligence. Saccades 2
Applications: Classification
Business•Credit rating and risk assessment •Insurance risk evaluation •Fraud detection •Insider dealing detection •Marketing analysis•Mailshot profiling •Signature verification •Inventory control
Engineering•Machinery defect diagnosis •Signal processing •Character recognition •Process supervision •Process fault analysis •Speech recognition •Machine vision •Speech recognition •Radar signal classification
Security•Face recognition •Speaker verification •Fingerprint analysis
Medicine•General diagnosis •Detection of heart defects
Science•Recognising genes •Botanical classification •Bacteria identification
![Page 19: Reading Assignmentsilab.usc.edu/classes/2002cs564/lecture_notes/17-Examples-Review.p… · FOn. Laurent Itti: CS564 - Brain Theory and Artificial Intelligence. Saccades 2 7 Neuron](https://reader033.fdocuments.net/reader033/viewer/2022060710/607699a78e6a3057b8035a97/html5/thumbnails/19.jpg)
19Laurent Itti: CS564 - Brain Theory and Artificial Intelligence. Saccades 2
Applications: Modelling
Business•Prediction of share and commodity prices •Prediction of economic indicators •Insider dealing detection •Marketing analysis•Mailshot profiling •Signature verification •Inventory control
Engineering•Transducer linerisation •Colour discrimination •Robot control and navigation •Process control •Aircraft landing control •Car active suspension control •Printed Circuit auto routing •Integrated circuit layout •Image compression
Science•Prediction of the performance of drugs from the molecular structure •Weather prediction •Sunspot prediction
Medicine•. Medical imaging and image processing
![Page 20: Reading Assignmentsilab.usc.edu/classes/2002cs564/lecture_notes/17-Examples-Review.p… · FOn. Laurent Itti: CS564 - Brain Theory and Artificial Intelligence. Saccades 2 7 Neuron](https://reader033.fdocuments.net/reader033/viewer/2022060710/607699a78e6a3057b8035a97/html5/thumbnails/20.jpg)
20Laurent Itti: CS564 - Brain Theory and Artificial Intelligence. Saccades 2
Applications: Forecasting
•Future sales •Production Requirements •Market Performance •Economic Indicators •Energy Requirements •Time Based Variables
![Page 21: Reading Assignmentsilab.usc.edu/classes/2002cs564/lecture_notes/17-Examples-Review.p… · FOn. Laurent Itti: CS564 - Brain Theory and Artificial Intelligence. Saccades 2 7 Neuron](https://reader033.fdocuments.net/reader033/viewer/2022060710/607699a78e6a3057b8035a97/html5/thumbnails/21.jpg)
21Laurent Itti: CS564 - Brain Theory and Artificial Intelligence. Saccades 2
Applications: Novelty Detection
•Fault Monitoring •Performance Monitoring •Fraud Detection •Detecting Rate Features •Different Cases
![Page 22: Reading Assignmentsilab.usc.edu/classes/2002cs564/lecture_notes/17-Examples-Review.p… · FOn. Laurent Itti: CS564 - Brain Theory and Artificial Intelligence. Saccades 2 7 Neuron](https://reader033.fdocuments.net/reader033/viewer/2022060710/607699a78e6a3057b8035a97/html5/thumbnails/22.jpg)
22Laurent Itti: CS564 - Brain Theory and Artificial Intelligence. Saccades 2
Multi-layer Perceptron Classifier
![Page 23: Reading Assignmentsilab.usc.edu/classes/2002cs564/lecture_notes/17-Examples-Review.p… · FOn. Laurent Itti: CS564 - Brain Theory and Artificial Intelligence. Saccades 2 7 Neuron](https://reader033.fdocuments.net/reader033/viewer/2022060710/607699a78e6a3057b8035a97/html5/thumbnails/23.jpg)
23Laurent Itti: CS564 - Brain Theory and Artificial Intelligence. Saccades 2
http://ams.egeo.sai.jrc.it/eurostat/Lot16-SUPCOM95/node7.html
Multi-layer Perceptron Classifier
![Page 24: Reading Assignmentsilab.usc.edu/classes/2002cs564/lecture_notes/17-Examples-Review.p… · FOn. Laurent Itti: CS564 - Brain Theory and Artificial Intelligence. Saccades 2 7 Neuron](https://reader033.fdocuments.net/reader033/viewer/2022060710/607699a78e6a3057b8035a97/html5/thumbnails/24.jpg)
24Laurent Itti: CS564 - Brain Theory and Artificial Intelligence. Saccades 2
Classifiers
http://www.electronicsletters.com/papers/2001/0020/paper.asp
1-stage approach
2-stageapproach
![Page 25: Reading Assignmentsilab.usc.edu/classes/2002cs564/lecture_notes/17-Examples-Review.p… · FOn. Laurent Itti: CS564 - Brain Theory and Artificial Intelligence. Saccades 2 7 Neuron](https://reader033.fdocuments.net/reader033/viewer/2022060710/607699a78e6a3057b8035a97/html5/thumbnails/25.jpg)
25Laurent Itti: CS564 - Brain Theory and Artificial Intelligence. Saccades 2
Example: face recognition
Here using the 2-stage approach:
![Page 26: Reading Assignmentsilab.usc.edu/classes/2002cs564/lecture_notes/17-Examples-Review.p… · FOn. Laurent Itti: CS564 - Brain Theory and Artificial Intelligence. Saccades 2 7 Neuron](https://reader033.fdocuments.net/reader033/viewer/2022060710/607699a78e6a3057b8035a97/html5/thumbnails/26.jpg)
26Laurent Itti: CS564 - Brain Theory and Artificial Intelligence. Saccades 2
Training
http://www.neci.nec.com/homepages/lawrence/papers/face-tr96/latex.html
![Page 27: Reading Assignmentsilab.usc.edu/classes/2002cs564/lecture_notes/17-Examples-Review.p… · FOn. Laurent Itti: CS564 - Brain Theory and Artificial Intelligence. Saccades 2 7 Neuron](https://reader033.fdocuments.net/reader033/viewer/2022060710/607699a78e6a3057b8035a97/html5/thumbnails/27.jpg)
27Laurent Itti: CS564 - Brain Theory and Artificial Intelligence. Saccades 2
Learning rate
![Page 28: Reading Assignmentsilab.usc.edu/classes/2002cs564/lecture_notes/17-Examples-Review.p… · FOn. Laurent Itti: CS564 - Brain Theory and Artificial Intelligence. Saccades 2 7 Neuron](https://reader033.fdocuments.net/reader033/viewer/2022060710/607699a78e6a3057b8035a97/html5/thumbnails/28.jpg)
28Laurent Itti: CS564 - Brain Theory and Artificial Intelligence. Saccades 2
Testing / Evaluation
Look at performance as a function of networkcomplexity
![Page 29: Reading Assignmentsilab.usc.edu/classes/2002cs564/lecture_notes/17-Examples-Review.p… · FOn. Laurent Itti: CS564 - Brain Theory and Artificial Intelligence. Saccades 2 7 Neuron](https://reader033.fdocuments.net/reader033/viewer/2022060710/607699a78e6a3057b8035a97/html5/thumbnails/29.jpg)
29Laurent Itti: CS564 - Brain Theory and Artificial Intelligence. Saccades 2
Testing / Evaluation
Comparison with other known techniques
![Page 30: Reading Assignmentsilab.usc.edu/classes/2002cs564/lecture_notes/17-Examples-Review.p… · FOn. Laurent Itti: CS564 - Brain Theory and Artificial Intelligence. Saccades 2 7 Neuron](https://reader033.fdocuments.net/reader033/viewer/2022060710/607699a78e6a3057b8035a97/html5/thumbnails/30.jpg)
30Laurent Itti: CS564 - Brain Theory and Artificial Intelligence. Saccades 2
Associative Memories
http://www.shef.ac.uk/psychology/gurney/notes/l5/l5.html
Idea: store:
So that we can recover it if presented with corrupted data such as:
![Page 31: Reading Assignmentsilab.usc.edu/classes/2002cs564/lecture_notes/17-Examples-Review.p… · FOn. Laurent Itti: CS564 - Brain Theory and Artificial Intelligence. Saccades 2 7 Neuron](https://reader033.fdocuments.net/reader033/viewer/2022060710/607699a78e6a3057b8035a97/html5/thumbnails/31.jpg)
31Laurent Itti: CS564 - Brain Theory and Artificial Intelligence. Saccades 2
Associative Memories
How can we set the weights such as to store multiplePatterns?
Use Hebbian learning!Result:
Wij =1/N Sum piu pj
u
trainingpatterns u
See HKP chapter 2.
![Page 32: Reading Assignmentsilab.usc.edu/classes/2002cs564/lecture_notes/17-Examples-Review.p… · FOn. Laurent Itti: CS564 - Brain Theory and Artificial Intelligence. Saccades 2 7 Neuron](https://reader033.fdocuments.net/reader033/viewer/2022060710/607699a78e6a3057b8035a97/html5/thumbnails/32.jpg)
32Laurent Itti: CS564 - Brain Theory and Artificial Intelligence. Saccades 2
Associative memory with Hopfield nets
Setup a Hopfield net such that local minima correspondto the stored patterns.
Issues:-because of weight symmetry, anti-patterns (binary reverse) are stored as well as the original patterns (also spurious local minima are created when many patterns are stored)
-if one tries to store more than about 0.14*(number of neurons)patterns, the network exhibits unstable behavior
- works well only if patterns are uncorrelated
![Page 33: Reading Assignmentsilab.usc.edu/classes/2002cs564/lecture_notes/17-Examples-Review.p… · FOn. Laurent Itti: CS564 - Brain Theory and Artificial Intelligence. Saccades 2 7 Neuron](https://reader033.fdocuments.net/reader033/viewer/2022060710/607699a78e6a3057b8035a97/html5/thumbnails/33.jpg)
33Laurent Itti: CS564 - Brain Theory and Artificial Intelligence. Saccades 2
Capabilities and Limitations of Layered Networks
Issues:
- what can given networks do?- What can they learn to do?- How many layers required for given task?- How many units per layer?- When will a network generalize?- What do we mean by generalize?- …
See HKP chapter 6.4.
![Page 34: Reading Assignmentsilab.usc.edu/classes/2002cs564/lecture_notes/17-Examples-Review.p… · FOn. Laurent Itti: CS564 - Brain Theory and Artificial Intelligence. Saccades 2 7 Neuron](https://reader033.fdocuments.net/reader033/viewer/2022060710/607699a78e6a3057b8035a97/html5/thumbnails/34.jpg)
34Laurent Itti: CS564 - Brain Theory and Artificial Intelligence. Saccades 2
Capabilities and Limitations of Layered Networks
What about boolean functions?
Single-layer perceptrons are very limited:- XOR problem- connectivity problem- etc.
But what about multilayer perceptrons?
We saw (midterm) that we can represent them with a network with just one hidden layer.
![Page 35: Reading Assignmentsilab.usc.edu/classes/2002cs564/lecture_notes/17-Examples-Review.p… · FOn. Laurent Itti: CS564 - Brain Theory and Artificial Intelligence. Saccades 2 7 Neuron](https://reader033.fdocuments.net/reader033/viewer/2022060710/607699a78e6a3057b8035a97/html5/thumbnails/35.jpg)
35Laurent Itti: CS564 - Brain Theory and Artificial Intelligence. Saccades 2
Capabilities and Limitations of Layered Networks
To approximate a set of functions of the inputs byA layered network with continuous-valued units andSigmoidal activation function…
Cybenko, 1988: … at most two hidden layers are necessary, with arbitrary accuracy attainable by adding more hidden units.
Cybenko, 1989: one hidden layer is enough to approximate any continuous function.
Intuition of proof: decompose function to be approximated into a sum of localized “bumps.” The bumps can be constructed with two hidden layers.
Similar in spirit to Fourier decomposition. Bumps = radial basisfunctions.
![Page 36: Reading Assignmentsilab.usc.edu/classes/2002cs564/lecture_notes/17-Examples-Review.p… · FOn. Laurent Itti: CS564 - Brain Theory and Artificial Intelligence. Saccades 2 7 Neuron](https://reader033.fdocuments.net/reader033/viewer/2022060710/607699a78e6a3057b8035a97/html5/thumbnails/36.jpg)
36Laurent Itti: CS564 - Brain Theory and Artificial Intelligence. Saccades 2
Optimal Network Architectures
How can we determine the number of hidden units?
- genetic algorithms: evaluate variations of the network, using a metric that combines its performance and its complexity. Then apply various mutations to the network (change number of hidden units) until the best one is found.
- Pruning and weight decay:- apply weight decay (remember reinforcement learning) during training- eliminate connections with weight below threshold- re-train
- How about eliminating units? For example, eliminate units with total synaptic input weight smaller than threshold.
![Page 37: Reading Assignmentsilab.usc.edu/classes/2002cs564/lecture_notes/17-Examples-Review.p… · FOn. Laurent Itti: CS564 - Brain Theory and Artificial Intelligence. Saccades 2 7 Neuron](https://reader033.fdocuments.net/reader033/viewer/2022060710/607699a78e6a3057b8035a97/html5/thumbnails/37.jpg)
37Laurent Itti: CS564 - Brain Theory and Artificial Intelligence. Saccades 2
For further information
See HKP:
Hertz, Krogh & Palmer: Introduction to the theory of neural computation (Addison Wesley)
In particular, the end of chapters 2 and 6.