Arti cial neural networks and other learning systems fileCourse outline Exam Introduction to ANNs...

History and use of ANNsCourse outline

ExamIntroduction to ANNs

Artificial neural networks and other learningsystems

Erik Fransen ANN



1 History and use of ANNsHistoryComponentsUse of ANNsNo free lunch

2 Course outlineLecturesLabsLab review

3 Exam

4 Introduction to ANNsCharacteristicsInspiration from biologyCurrent trends

Erik Fransen ANN



HistoryComponentsUse of ANNsNo free lunch



3 Exam


Erik Fransen ANN




History

Artificial neural network, connectionist network, neurocomputingneuronnat, neuralt nat

Early 40’s first nets1969 Minsky and Paperts critisizm80’s revival with Hopfield and Backprop90’s incorporated into Machine Learning

Early neuromime, Bell

Labs. One neuron with

5 excitatory inputs and

1 inhibitory input, 1

output.

BellCore chip. 32

neurons, 496 synapses,

100.000 patterns/ sec.

Erik Fransen ANN




Components

topology

nodes

activation function

learning rule

assumptions on data in

type of data out

Erik Fransen ANN




Use of ANNs

Classification

Pattern recognition

Diagnosis

Signal processing

Data coding

Clustering

Input-output mapping (associator)

Interpolation/generalization

Process modeling

Optimization with soft constraints

Erik Fransen ANN




No free lunch

The no free lunch theorem states that averaging over all possibleworlds (data sets/problems), there is no single optimal algorithm.But, given a specific domain some algorthms performs better thanother.

Get to know your data by plotting it, do standardstatistical analysis (measure mean, standard deviation,PCA, etc)

Understand what you want to accomplish, what is yourdesired output

Set up your strategy, steps of analysis

Erik Fransen ANN



LecturesLabsLab review



3 Exam


Erik Fransen ANN




Lectures

Feed-forward networks, Supervised learningframatkopplat nat, lararledd/overvakad inlarning

2 One layer perceptron

3 Multi layer perceptron

4 Multi layer perceptron, cont. (L1), Generalization, thesuport vector machine

Erik Fransen ANN




Lectures

Feed-forward, Unsupervised learning/Self-organizationoovervakad inlarning, sjalvorganisation

5 Principal component analysis, independent componentanalysis (L2)

6 Self organizing maps, vector quantization (L3)

Erik Fransen ANN




Lectures

Feed-back networks, recurrent networks, supervisedaterkopplade nat

7 Boltzmann machines, Hopfield nets (L4)

8 Processing of temporal data, echo state networks

Erik Fransen ANN




Lectures

Domain assumptions, representation etc.

8 Ensemble techniques (boosting, bagging)

9 Regularization, radial basis functions

10 Domain assumptions, representation, inversemodeling, Reinforcement learning

11 Your questions. Old exam questions

Erik Fransen ANN




Labs

Feed forward networks - Delta rule and Backpropagation(m+p)

Competetive learning, coding with radial basis functions(m)

Self organizing feature maps (m+p)

Hopfield networks (m+p)

Erik Fransen ANN




Lab review

Lab review (labbredovisning)In the lab review you have 10 minutes to convince the instructorthat you should pass the lab exam. Be careful about planning thereview so that you use the time well.You are in charge of the lab review, but we may control who talks(each participant must have knowledge of all parts of the review)and we may ask for clearifications if that is necessary.If you have points that are unclear about the lab, bring that upduring the help time and not during the review. The review is notbeside a computer, so bring all necessary material in terms ofgraphs etc.

Erik Fransen ANN




Lab review

Lab review (cont’d)Review main pointsWhat is the lab about? What are the main points of the lab?What are your conclusions based on these main points?What must be known before the lab work can be started?What did you learn by doing the lab?Briefly describe the lab result main points.Show results in terms of calculations, figures, tables etc.For each result, what are your conclusions?Any final questions based on your results and conclusions?

Erik Fransen ANN



Exam

material from lectures, books, labs

grade A-F

look at old exams

know concepts and words, a bit of proofs

simple exercises

geometric/graphical understanding of how it works

how to select algorithm given a problem

Erik Fransen ANN



CharacteristicsInspiration from biologyCurrent trends



3 Exam


Erik Fransen ANN




Characteristics

nodes

(enhet, nod)

activation function

(aktiveringsfunktion)

learning rule

(inlarningsregel)

topology

(topologi)


type of data out

ANN node

w

w

w

w

Σ ϕ

input

output

Erik Fransen ANN




Characteristics

nodes

activation function

learning rule

topology


type of data out

activationfunction

Σ

ϕ

Σ

ϕ

Σ

ϕ

Σ

ϕ

piece wise linear sigmoid

linear threshold

Erik Fransen ANN




Characteristics

nodes

activation function

learning rule

topology


type of data out

supervisedin

∆

desired out

correction

w out

Erik Fransen ANN




Characteristics

nodes

activation function

learning rule

topology


type of data out

unsupervised,competitive

out

Erik Fransen ANN




Characteristics

nodes

activation function

learning rule

topology


type of data out

Hebbian,co-active,correlated

0

1

0

1

+w

Erik Fransen ANN




Characteristics

nodes

activation function

learning rule

topology


type of data out

one layerfeed-forward

Erik Fransen ANN




Characteristics

nodes

activation function

learning rule

topology


type of data out

two layersfeed-forward

Erik Fransen ANN




Characteristics

nodes

activation function

learning rule

topology


type of data out

feed-back,recurrent

Erik Fransen ANN




Inspiration from biology

The neuron

Erik Fransen ANN





nerve impulse, “spikes”

threshold activation all-or-nothing

travels along the axon

fixed amplitude

activates synapses

The actionpotential

Erik Fransen ANN





spike releases transmitter

transmitter activates receptor

receptor opens channel

channel gives excitatory EPSP

channel gives inhibitory IPSP

cascade changes during learning

EPSPs andIPSPs

Erik Fransen ANN




Biological neurons

many simple neurons - nodes

the nerve impulse is all-or-nothing - 1 or 0 out

many inputs and outputs per neuron - many connections

analog transmission in synapse - weight in connection

input summed in cell body - summation of input

sum determines action potential frequency - transfer fkn

Erik Fransen ANN




Biological neurons

information processing based on local information

memory stored in weights

weights are changed according to learning rules

weights can be positive and negative

parallell processing is fast (but learning takes time)

tolerance to errors in input, in weights and in outputs

Erik Fransen ANN




Current trends

Machine learning

Mathematical statistics

Information theory

Computability theory

But see also brain inspired algorithms like that of Jeff Hawkins(founder of Palm Computing) and his company Numenta.

Erik Fransen ANN




Current use of ANNs

Current contests where ANNs probably will show upCompleted contests ...Some examples of applicationsExamples of companies and projectsExamples of masters thesis (exjobb) ...Conferences about ANNs and learning systemsOrganizations

See also our course DD2431 Machine learning, 6hp, per1.

Erik Fransen ANN

Arti cial neural networks and other learning systems fileCourse outline Exam Introduction to ANNs...

Documents

Transcript of Arti cial neural networks and other learning systems fileCourse outline Exam Introduction to ANNs...