Neural NetworksChapter 2

Joost N. Kok

Universiteit Leiden

Hopfield Networks

• Network of McCulloch-Pitts neurons • Output is 1 iff and is -1

otherwise i

jjijSw

Hopfield Networks

Hopfield Networks

Hopfield Networks

Hopfield Networks

• Associative Memory Problem:

Store a set of patterns in such a way that when presented with a new pattern, the network responds by producing whichever of the stored patterns most closely resembles the new pattern.

Hopfield Networks

• Resembles = Hamming distance

• Configuration space = all possible states of the network

• Stored patterns should be attractors

• Basins of attractors

Hopfield Networks

• N neurons

• Two states: -1 (silent) and 1 (firing)

• Fully connected

• Symmetric Weights

• Thresholds

Hopfield Networksw13

w16 w57

-1 +1

Hopfield Networks

• State:

• Weights:

• Dynamics:

25,251,25

25,11,1

ww

ww

w

251 ... SSS

25

1sgn:

i jiji SwS

Hopfield Networks

• Hebb’s learning rule: – Make connection stronger if neurons have the

same state– Make connection weaker if the neurons have a

different state

Hopfield Networksneuron 1 synapse neuron 2

)(

1

)(1

j

p

ipijw

Hopfield Networks

• Weight between neuron i and neuron j is given by

Hopfield Networks

• Opposite patterns give the same weights

• This implies that they are also stable points of the network

• Capacity of Hopfield Networks is limited: 0.14 N

Hopfield Networks

• Hopfield defines the energy of a network:

E = - ½ ij SiSjwij + i Sii

• If we pick unit i and the firing rule does not change its Si, it will not change E.

• If we pick unit i and the firing rule does change its Si, it will decrease E.

Hopfield Networks

• Energy function:

• Alternative Form:

• Updates:

ij

jiij SSwH2

1

)(ij

jiij SSwCH

jjiji SwS sgn'

Hopfield Networks

ij

jiijij

jiij SSwSSwHH ''

iij

jijiij

jiji wSwSSwSHH 222'

Hopfield Networks

• Extension: use stochastic fire rule– Si := +1 with probability g(hi)– Si := -1 with probability 1-g(hi)

Hopfield Networks

• Nonlinear function:

x

g(x)

g(x) = 1 + e – x

1

0

Hopfield Networks• Replace the binary threshold units by binary stochastic

units.• Define = 1/T• Use “temperature” T to make it easier to cross energy

barriers.– Start at high temperature where its easy to cross energy barriers.

– Reduce slowly to low temperature where good states are much more probable than bad ones.

A B C

Hopfield Networks

• Kick the network our of spurious local minima

• Equilibrium: becomes time independent

iS

)..2exp(1

11Pr

iii h

hfS

T

1