o E v } o Ì u v v D Z ] v · 2021. 3. 5. · d Z , o u Z } o Ì & v P Ç } ( ^ Ç u Ç u Z o ] ( ]...

104
Neural Networks Hopfield Nets and Boltzmann Machines 1

Transcript of o E v } o Ì u v v D Z ] v · 2021. 3. 5. · d Z , o u Z } o Ì & v P Ç } ( ^ Ç u Ç u Z o ] ( ]...

Page 1: o E v } o Ì u v v D Z ] v · 2021. 3. 5. · d Z , o u Z } o Ì & v P Ç } ( ^ Ç u Ç u Z o ] ( ] u µ v v o Ç À Ç ] v P Z Á Z ] Z ] À ] ] Z À ] } µ U }

Neural Networks

Hopfield Nets and Boltzmann Machines

1

Page 2: o E v } o Ì u v v D Z ] v · 2021. 3. 5. · d Z , o u Z } o Ì & v P Ç } ( ^ Ç u Ç u Z o ] ( ] u µ v v o Ç À Ç ] v P Z Á Z ] Z ] À ] ] Z À ] } µ U }

Recap: Hopfield network

• At each time each neuron receives a “field”

• If the sign of the field matches its own sign, it does not respond

• If the sign of the field opposes its own sign, it “flips” to match the sign of the field

2

Page 3: o E v } o Ì u v v D Z ] v · 2021. 3. 5. · d Z , o u Z } o Ì & v P Ç } ( ^ Ç u Ç u Z o ] ( ] u µ v v o Ç À Ç ] v P Z Á Z ] Z ] À ] ] Z À ] } µ U }

Recap: Energy of a Hopfield Network

,

• The system will evolve until the energy hits a local minimum• In vector form

– Bias term may be viewed as an extra input pegged to 1.0

3

Page 4: o E v } o Ì u v v D Z ] v · 2021. 3. 5. · d Z , o u Z } o Ì & v P Ç } ( ^ Ç u Ç u Z o ] ( ] u µ v v o Ç À Ç ] v P Z Á Z ] Z ] À ] ] Z À ] } µ U }

Recap: Hopfield net computation

• Very simple• Updates can be done sequentially, or all at once• Convergence

does not change significantly any more

1. Initialize network with initial pattern

2. Iterate until convergence

4

Page 5: o E v } o Ì u v v D Z ] v · 2021. 3. 5. · d Z , o u Z } o Ì & v P Ç } ( ^ Ç u Ç u Z o ] ( ] u µ v v o Ç À Ç ] v P Z Á Z ] Z ] À ] ] Z À ] } µ U }

Recap: Evolution

• The network will evolve until it arrives at a local minimum in the energy contour

statePE

5

Page 6: o E v } o Ì u v v D Z ] v · 2021. 3. 5. · d Z , o u Z } o Ì & v P Ç } ( ^ Ç u Ç u Z o ] ( ] u µ v v o Ç À Ç ] v P Z Á Z ] Z ] À ] ] Z À ] } µ U }

Recap: Content-addressable memory

• Each of the minima is a “stored” pattern– If the network is initialized close to a stored pattern, it

will inevitably evolve to the pattern

• This is a content addressable memory– Recall memory content from partial or corrupt values

• Also called associative memory

statePE

6

Page 7: o E v } o Ì u v v D Z ] v · 2021. 3. 5. · d Z , o u Z } o Ì & v P Ç } ( ^ Ç u Ç u Z o ] ( ] u µ v v o Ç À Ç ] v P Z Á Z ] Z ] À ] ] Z À ] } µ U }

Examples: Content addressable memory

• http://staff.itee.uq.edu.au/janetw/cmc/chapters/Hopfield/ 7

Page 8: o E v } o Ì u v v D Z ] v · 2021. 3. 5. · d Z , o u Z } o Ì & v P Ç } ( ^ Ç u Ç u Z o ] ( ] u µ v v o Ç À Ç ] v P Z Á Z ] Z ] À ] ] Z À ] } µ U }

Examples: Content addressable memory

• http://staff.itee.uq.edu.au/janetw/cmc/chapters/Hopfield/ 8

Noisy pattern completion: Initialize the entire network and let the entire network evolve

Page 9: o E v } o Ì u v v D Z ] v · 2021. 3. 5. · d Z , o u Z } o Ì & v P Ç } ( ^ Ç u Ç u Z o ] ( ] u µ v v o Ç À Ç ] v P Z Á Z ] Z ] À ] ] Z À ] } µ U }

Examples: Content addressable memory

• http://staff.itee.uq.edu.au/janetw/cmc/chapters/Hopfield/ 9

Pattern completion: Fix the “seen” bits and onlylet the “unseen” bits evolve

Page 10: o E v } o Ì u v v D Z ] v · 2021. 3. 5. · d Z , o u Z } o Ì & v P Ç } ( ^ Ç u Ç u Z o ] ( ] u µ v v o Ç À Ç ] v P Z Á Z ] Z ] À ] ] Z À ] } µ U }

Training a Hopfield Net to “Memorize” target patterns

• The Hopfield network can be trained to remember specific “target” patterns– E.g. the pictures in the previous example

• This can be done by setting the weights appropriately

10

Random Question:Can you use backprop to train Hopfield nets?

Hint: Think RNN

Page 11: o E v } o Ì u v v D Z ] v · 2021. 3. 5. · d Z , o u Z } o Ì & v P Ç } ( ^ Ç u Ç u Z o ] ( ] u µ v v o Ç À Ç ] v P Z Á Z ] Z ] À ] ] Z À ] } µ U }

Training a Hopfield Net to “Memorize” target patterns

• The Hopfield network can be trained to remember specific “target” patterns– E.g. the pictures in the previous example

• A Hopfield net with neurons can designed to store up to target -bit memories– But can store an exponential number of unwanted “parasitic”

memories along with the target patterns

• Training the network: Design weights matrix such that the energy of … – Target patterns is minimized, so that they are in energy wells– Other untargeted potentially parasitic patterns is maximized so that

they don’t become parasitic11

Page 12: o E v } o Ì u v v D Z ] v · 2021. 3. 5. · d Z , o u Z } o Ì & v P Ç } ( ^ Ç u Ç u Z o ] ( ] u µ v v o Ç À Ç ] v P Z Á Z ] Z ] À ] ] Z À ] } µ U }

Training the network

12state

Energy

Minimize energy of target patterns

Maximize energy of all other patterns

Page 13: o E v } o Ì u v v D Z ] v · 2021. 3. 5. · d Z , o u Z } o Ì & v P Ç } ( ^ Ç u Ç u Z o ] ( ] u µ v v o Ç À Ç ] v P Z Á Z ] Z ] À ] ] Z À ] } µ U }

Optimizing W

• Simple gradient descent:

Minimize energy of target patterns

Maximize energy of all other patterns

Page 14: o E v } o Ì u v v D Z ] v · 2021. 3. 5. · d Z , o u Z } o Ì & v P Ç } ( ^ Ç u Ç u Z o ] ( ] u µ v v o Ç À Ç ] v P Z Á Z ] Z ] À ] ] Z À ] } µ U }

Training the network

14state

Energy

Minimize energy of target patterns

Maximize energy of all other patterns

Page 15: o E v } o Ì u v v D Z ] v · 2021. 3. 5. · d Z , o u Z } o Ì & v P Ç } ( ^ Ç u Ç u Z o ] ( ] u µ v v o Ç À Ç ] v P Z Á Z ] Z ] À ] ] Z À ] } µ U }

Simpler: Focus on confusing parasites

• Focus on minimizing parasites that can prevent the net from remembering target patterns– Energy valleys in the neighborhood of target patterns

15state

Energy

Page 16: o E v } o Ì u v v D Z ] v · 2021. 3. 5. · d Z , o u Z } o Ì & v P Ç } ( ^ Ç u Ç u Z o ] ( ] u µ v v o Ç À Ç ] v P Z Á Z ] Z ] À ] ] Z À ] } µ U }

Training to maximize memorability of target patterns

16state

Energy

• Lower energy at valid memories• Initialize the network at valid memories and let it evolve

– It will settle in a valley. If this is not the target pattern, raise it

Page 17: o E v } o Ì u v v D Z ] v · 2021. 3. 5. · d Z , o u Z } o Ì & v P Ç } ( ^ Ç u Ç u Z o ] ( ] u µ v v o Ç À Ç ] v P Z Á Z ] Z ] À ] ] Z À ] } µ U }

Training the Hopfield network

• Initialize • Compute the total outer product of all target patterns

– More important patterns presented more frequently

• Initialize the network with each target pattern and let it evolve– And settle at a valley

• Compute the total outer product of valley patterns• Update weights

17

Page 18: o E v } o Ì u v v D Z ] v · 2021. 3. 5. · d Z , o u Z } o Ì & v P Ç } ( ^ Ç u Ç u Z o ] ( ] u µ v v o Ç À Ç ] v P Z Á Z ] Z ] À ] ] Z À ] } µ U }

Training the Hopfield network: SGD version

• Initialize • Do until convergence, satisfaction, or death from

boredom:– Sample a target pattern

• Sampling frequency of pattern must reflect importance of pattern

– Initialize the network at and let it evolve• And settle at a valley

– Update weights•

18

Page 19: o E v } o Ì u v v D Z ] v · 2021. 3. 5. · d Z , o u Z } o Ì & v P Ç } ( ^ Ç u Ç u Z o ] ( ] u µ v v o Ç À Ç ] v P Z Á Z ] Z ] À ] ] Z À ] } µ U }

More efficient training• Really no need to raise the entire surface, or even

every valley• Raise the neighborhood of each target memory

– Sufficient to make the memory a valley– The broader the neighborhood considered, the

broader the valley

19state

Energy

Page 20: o E v } o Ì u v v D Z ] v · 2021. 3. 5. · d Z , o u Z } o Ì & v P Ç } ( ^ Ç u Ç u Z o ] ( ] u µ v v o Ç À Ç ] v P Z Á Z ] Z ] À ] ] Z À ] } µ U }

Training the Hopfield network: SGD version

• Initialize • Do until convergence, satisfaction, or death from

boredom:– Sample a target pattern

• Sampling frequency of pattern must reflect importance of pattern

– Initialize the network at and let it evolve a few steps (2-4)• And arrive at a down-valley position

– Update weights•

20

Page 21: o E v } o Ì u v v D Z ] v · 2021. 3. 5. · d Z , o u Z } o Ì & v P Ç } ( ^ Ç u Ç u Z o ] ( ] u µ v v o Ç À Ç ] v P Z Á Z ] Z ] À ] ] Z À ] } µ U }

Problem with Hopfield net

• Why is the recalled pattern not perfect?

21

Page 22: o E v } o Ì u v v D Z ] v · 2021. 3. 5. · d Z , o u Z } o Ì & v P Ç } ( ^ Ç u Ç u Z o ] ( ] u µ v v o Ç À Ç ] v P Z Á Z ] Z ] À ] ] Z À ] } µ U }

A Problem with Hopfield Nets

• Many local minima– Parasitic memories

• May be escaped by adding some noise during evolution– Permit changes in state even if energy increases..

• Particularly if the increase in energy is small22

state

Energy

Parasitic memories

Page 23: o E v } o Ì u v v D Z ] v · 2021. 3. 5. · d Z , o u Z } o Ì & v P Ç } ( ^ Ç u Ç u Z o ] ( ] u µ v v o Ç À Ç ] v P Z Á Z ] Z ] À ] ] Z À ] } µ U }

Recap – Analogy: Spin Glasses

• The total energy of the system

• The system evolves to minimize the energy– Dipoles stop flipping if flips result in increase of energy

Total field at current dipole:

Response of current diplose

23

Page 24: o E v } o Ì u v v D Z ] v · 2021. 3. 5. · d Z , o u Z } o Ì & v P Ç } ( ^ Ç u Ç u Z o ] ( ] u µ v v o Ç À Ç ] v P Z Á Z ] Z ] À ] ] Z À ] } µ U }

Recap : Spin Glasses

• The system stops at one of its stable configurations– Where energy is a local minimum

state

PE

24

Page 25: o E v } o Ì u v v D Z ] v · 2021. 3. 5. · d Z , o u Z } o Ì & v P Ç } ( ^ Ç u Ç u Z o ] ( ] u µ v v o Ç À Ç ] v P Z Á Z ] Z ] À ] ] Z À ] } µ U }

Revisiting Thermodynamic Phenomena

• Is the system actually in a specific state at any time?• No – the state is actually continuously changing

– Based on the temperature of the system• At higher temperatures, state changes more rapidly

• What is actually being characterized is the probability of the state at equilibrium– The system “prefers” low energy states– Evolution of the system favors transitions towards lower-energy states

state

PE

Page 26: o E v } o Ì u v v D Z ] v · 2021. 3. 5. · d Z , o u Z } o Ì & v P Ç } ( ^ Ç u Ç u Z o ] ( ] u µ v v o Ç À Ç ] v P Z Á Z ] Z ] À ] ] Z À ] } µ U }

The Helmholtz Free Energy of a System

• A thermodynamic system at temperature can exist in one of many states– Potentially infinite states– At any time, the probability of finding the system in state

at temperature is

• At each state it has a potential energy • The internal energy of the system, representing its

capacity to do work, is the average:

Page 27: o E v } o Ì u v v D Z ] v · 2021. 3. 5. · d Z , o u Z } o Ì & v P Ç } ( ^ Ç u Ç u Z o ] ( ] u µ v v o Ç À Ç ] v P Z Á Z ] Z ] À ] ] Z À ] } µ U }

The Helmholtz Free Energy of a System

• The capacity to do work is counteracted by the internal disorder of the system, i.e. its entropy

• The Helmholtz free energy of the system measures the useful work derivable from it and combines the two terms

Page 28: o E v } o Ì u v v D Z ] v · 2021. 3. 5. · d Z , o u Z } o Ì & v P Ç } ( ^ Ç u Ç u Z o ] ( ] u µ v v o Ç À Ç ] v P Z Á Z ] Z ] À ] ] Z À ] } µ U }

The Helmholtz Free Energy of a System

• A system held at a specific temperature anneals by varying the rate at which it visits the various states, to reduce the free energy in the system, until a minimum free-energy state is achieved

• The probability distribution of the states at steady state is known as the Boltzmann distribution

Page 29: o E v } o Ì u v v D Z ] v · 2021. 3. 5. · d Z , o u Z } o Ì & v P Ç } ( ^ Ç u Ç u Z o ] ( ] u µ v v o Ç À Ç ] v P Z Á Z ] Z ] À ] ] Z À ] } µ U }

The Helmholtz Free Energy of a System

• Minimizing this w.r.t , we get

– Also known as the Gibbs distribution– is a normalizing constant– Note the dependence on – A = 0, the system will always remain at the lowest-

energy configuration with prob = 1.

Page 30: o E v } o Ì u v v D Z ] v · 2021. 3. 5. · d Z , o u Z } o Ì & v P Ç } ( ^ Ç u Ç u Z o ] ( ] u µ v v o Ç À Ç ] v P Z Á Z ] Z ] À ] ] Z À ] } µ U }

Revisiting Thermodynamic Phenomena

• The evolution of the system is actually stochastic• At equilibrium the system visits various states according to

the Boltzmann distribution– The probability of any state is inversely related to its energy

• and also temperatures:

• The most likely state is the lowest energy state

state

PE

Page 31: o E v } o Ì u v v D Z ] v · 2021. 3. 5. · d Z , o u Z } o Ì & v P Ç } ( ^ Ç u Ç u Z o ] ( ] u µ v v o Ç À Ç ] v P Z Á Z ] Z ] À ] ] Z À ] } µ U }

Returning to the problem with Hopfield Nets

• Many local minima– Parasitic memories

• May be escaped by adding some noise during evolution– Permit changes in state even if energy increases..

• Particularly if the increase in energy is small31

state

Energy

Parasitic memories

Page 32: o E v } o Ì u v v D Z ] v · 2021. 3. 5. · d Z , o u Z } o Ì & v P Ç } ( ^ Ç u Ç u Z o ] ( ] u µ v v o Ç À Ç ] v P Z Á Z ] Z ] À ] ] Z À ] } µ U }

The Hopfield net as a distribution

• Mimics the Spin glass system

• The stochastic Hopfield network models a probability distribution over states– Where a state is a binary string– Specifically, it models a Boltzmann distribution– The parameters of the model are the weights of the network

• The probability that (at equilibrium) the network will be in any state is – It is a generative model: generates states according to

Visible Neurons

Page 33: o E v } o Ì u v v D Z ] v · 2021. 3. 5. · d Z , o u Z } o Ì & v P Ç } ( ^ Ç u Ç u Z o ] ( ] u µ v v o Ç À Ç ] v P Z Á Z ] Z ] À ] ] Z À ] } µ U }

The field at a single node• Let and be otherwise identical states that only differ in the i-th bit

– S has i-th bit = and S’ has i-th bit =

33

Page 34: o E v } o Ì u v v D Z ] v · 2021. 3. 5. · d Z , o u Z } o Ì & v P Ç } ( ^ Ç u Ç u Z o ] ( ] u µ v v o Ç À Ç ] v P Z Á Z ] Z ] À ] ] Z À ] } µ U }

The field at a single node

• Let and be the states with the ith bit in the and states

•

34

Page 35: o E v } o Ì u v v D Z ] v · 2021. 3. 5. · d Z , o u Z } o Ì & v P Ç } ( ^ Ç u Ç u Z o ] ( ] u µ v v o Ç À Ç ] v P Z Á Z ] Z ] À ] ] Z À ] } µ U }

The field at a single node

• Giving us

• The probability of any node taking value 1 given other node values is a logistic

35

Page 36: o E v } o Ì u v v D Z ] v · 2021. 3. 5. · d Z , o u Z } o Ì & v P Ç } ( ^ Ç u Ç u Z o ] ( ] u µ v v o Ç À Ç ] v P Z Á Z ] Z ] À ] ] Z À ] } µ U }

Redefining the network

• First try: Redefine a regular Hopfield net as a stochastic system• Each neuron is now a stochastic unit with a binary state , which

can take value 0 or 1 with a probability that depends on the local field– Note the slight change from Hopfield nets– Not actually necessary; only a matter of convenience

Visible Neurons

Page 37: o E v } o Ì u v v D Z ] v · 2021. 3. 5. · d Z , o u Z } o Ì & v P Ç } ( ^ Ç u Ç u Z o ] ( ] u µ v v o Ç À Ç ] v P Z Á Z ] Z ] À ] ] Z À ] } µ U }

The Hopfield net is a distribution

• The Hopfield net is a probability distribution over binary sequences– The Boltzmann distribution

• The conditional distribution of individual bits in the sequence is a logistic

Visible Neurons

Page 38: o E v } o Ì u v v D Z ] v · 2021. 3. 5. · d Z , o u Z } o Ì & v P Ç } ( ^ Ç u Ç u Z o ] ( ] u µ v v o Ç À Ç ] v P Z Á Z ] Z ] À ] ] Z À ] } µ U }

Running the network

• Initialize the neurons• Cycle through the neurons and randomly set the neuron to 1 or 0 according to the

probability given above– Gibbs sampling: Fix N-1 variables and sample the remaining variable– As opposed to energy-based update (mean field approximation): run the test zi > 0 ?

• After many many iterations (until “convergence”), sample the individual neurons

Visible Neurons

Page 39: o E v } o Ì u v v D Z ] v · 2021. 3. 5. · d Z , o u Z } o Ì & v P Ç } ( ^ Ç u Ç u Z o ] ( ] u µ v v o Ç À Ç ] v P Z Á Z ] Z ] À ] ] Z À ] } µ U }

Recap: Stochastic Hopfield Nets

• The evolution of the Hopfield net can be made stochastic

• Instead of deterministically responding to the sign of the local field, each neuron responds probabilistically– This is much more in accord with Thermodynamic models– The evolution of the network is more likely to escape spurious

“weak” memories39

Page 40: o E v } o Ì u v v D Z ] v · 2021. 3. 5. · d Z , o u Z } o Ì & v P Ç } ( ^ Ç u Ç u Z o ] ( ] u µ v v o Ç À Ç ] v P Z Á Z ] Z ] À ] ] Z À ] } µ U }

Recap: Stochastic Hopfield Nets

• The evolution of the Hopfield net can be made stochastic

• Instead of deterministically responding to the sign of the local field, each neuron responds probabilistically– This is much more in accord with Thermodynamic models– The evolution of the network is more likely to escape spurious

“weak” memories40

The field quantifies the energy difference obtained by flipping thecurrent unit

Page 41: o E v } o Ì u v v D Z ] v · 2021. 3. 5. · d Z , o u Z } o Ì & v P Ç } ( ^ Ç u Ç u Z o ] ( ] u µ v v o Ç À Ç ] v P Z Á Z ] Z ] À ] ] Z À ] } µ U }

Recap: Stochastic Hopfield Nets

• The evolution of the Hopfield net can be made stochastic

• Instead of deterministically responding to the sign of the local field, each neuron responds probabilistically– This is much more in accord with Thermodynamic models– The evolution of the network is more likely to escape spurious

“weak” memories41

If the difference is not large, the probability of flipping approaches 0.5

The field quantifies the energy difference obtained by flipping thecurrent unit

Page 42: o E v } o Ì u v v D Z ] v · 2021. 3. 5. · d Z , o u Z } o Ì & v P Ç } ( ^ Ç u Ç u Z o ] ( ] u µ v v o Ç À Ç ] v P Z Á Z ] Z ] À ] ] Z À ] } µ U }

Recap: Stochastic Hopfield Nets

• The evolution of the Hopfield net can be made stochastic

• Instead of deterministically responding to the sign of the local field, each neuron responds probabilistically– This is much more in accord with Thermodynamic models– The evolution of the network is more likely to escape spurious

“weak” memories42

If the difference is not large, the probability of flipping approaches 0.5

The field quantifies the energy difference obtained by flipping thecurrent unit

T is a “temperature” parameter: increasing it moves the probability of thebits towards 0.5At T=1.0 we get the traditional definition of field and energyAt T = 0, we get deterministic Hopfield behavior

Page 43: o E v } o Ì u v v D Z ] v · 2021. 3. 5. · d Z , o u Z } o Ì & v P Ç } ( ^ Ç u Ç u Z o ] ( ] u µ v v o Ç À Ç ] v P Z Á Z ] Z ] À ] ] Z À ] } µ U }

Evolution of a stochastic Hopfield net

1. Initialize network with initial pattern

2. Iterate

43

Assuming T = 1

Page 44: o E v } o Ì u v v D Z ] v · 2021. 3. 5. · d Z , o u Z } o Ì & v P Ç } ( ^ Ç u Ç u Z o ] ( ] u µ v v o Ç À Ç ] v P Z Á Z ] Z ] À ] ] Z À ] } µ U }

Evolution of a stochastic Hopfield net

• When do we stop?• What is the final state of the system

– How do we “recall” a memory?

1. Initialize network with initial pattern

2. Iterate

44

Assuming T = 1

Page 45: o E v } o Ì u v v D Z ] v · 2021. 3. 5. · d Z , o u Z } o Ì & v P Ç } ( ^ Ç u Ç u Z o ] ( ] u µ v v o Ç À Ç ] v P Z Á Z ] Z ] À ] ] Z À ] } µ U }

Evolution of a stochastic Hopfield net

• When do we stop?• What is the final state of the system

– How do we “recall” a memory?

1. Initialize network with initial pattern

2. Iterate

45

Assuming T = 1

Page 46: o E v } o Ì u v v D Z ] v · 2021. 3. 5. · d Z , o u Z } o Ì & v P Ç } ( ^ Ç u Ç u Z o ] ( ] u µ v v o Ç À Ç ] v P Z Á Z ] Z ] À ] ] Z À ] } µ U }

Evolution of a stochastic Hopfield net

• Let the system evolve to “equilibrium”• Let be the sequence of values ( large)• Final predicted configuration: from the average of the final few iterations

– Estimates the probability that the bit is 1.0. – If it is greater than 0.5, sets it to 1.0

1. Initialize network with initial pattern

2. Iterate

46

Assuming T = 1

Page 47: o E v } o Ì u v v D Z ] v · 2021. 3. 5. · d Z , o u Z } o Ì & v P Ç } ( ^ Ç u Ç u Z o ] ( ] u µ v v o Ç À Ç ] v P Z Á Z ] Z ] À ] ] Z À ] } µ U }

Annealing

• Let the system evolve to “equilibrium”• Let be the sequence of values ( large)• Final predicted configuration: from the average of the final few iterations

1. Initialize network with initial pattern

2. For

i. For iter

a) For

47

Page 48: o E v } o Ì u v v D Z ] v · 2021. 3. 5. · d Z , o u Z } o Ì & v P Ç } ( ^ Ç u Ç u Z o ] ( ] u µ v v o Ç À Ç ] v P Z Á Z ] Z ] À ] ] Z À ] } µ U }

Evolution of the stochastic network

• Let the system evolve to “equilibrium”• Let be the sequence of values ( large)• Final predicted configuration: from the average of the final few iterations

1. Initialize network with initial pattern

2. For i. For iter

a) For

48

Pattern completion: Fix the “seen” bits and onlylet the “unseen” bits evolve

Noisy pattern completion: Initialize the entire network and let the entire network evolve

Page 49: o E v } o Ì u v v D Z ] v · 2021. 3. 5. · d Z , o u Z } o Ì & v P Ç } ( ^ Ç u Ç u Z o ] ( ] u µ v v o Ç À Ç ] v P Z Á Z ] Z ] À ] ] Z À ] } µ U }

Evolution of a stochastic Hopfield net

• When do we stop?• What is the final state of the system

– How do we “recall” a memory?

1. Initialize network with initial pattern

2. Iterate

49

Assuming T = 1

Page 50: o E v } o Ì u v v D Z ] v · 2021. 3. 5. · d Z , o u Z } o Ì & v P Ç } ( ^ Ç u Ç u Z o ] ( ] u µ v v o Ç À Ç ] v P Z Á Z ] Z ] À ] ] Z À ] } µ U }

Recap: Stochastic Hopfield Nets

• The probability of each neuron is given by a conditional distribution

• What is the overall probability of the entire set of neurons taking any configuration

50

Page 51: o E v } o Ì u v v D Z ] v · 2021. 3. 5. · d Z , o u Z } o Ì & v P Ç } ( ^ Ç u Ç u Z o ] ( ] u µ v v o Ç À Ç ] v P Z Á Z ] Z ] À ] ] Z À ] } µ U }

The overall probability

• The probability of any state can be shown to be given by the Boltzmann distribution

– Minimizing energy maximizes log likelihood51

Page 52: o E v } o Ì u v v D Z ] v · 2021. 3. 5. · d Z , o u Z } o Ì & v P Ç } ( ^ Ç u Ç u Z o ] ( ] u µ v v o Ç À Ç ] v P Z Á Z ] Z ] À ] ] Z À ] } µ U }

The Hopfield net is a distribution

• The Hopfield net is a probability distribution over binary sequences– The Boltzmann distribution

– The parameter of the distribution is the weights matrix

• The conditional distribution of individual bits in the sequence is a logistic• We will call this a Boltzmann machine

Page 53: o E v } o Ì u v v D Z ] v · 2021. 3. 5. · d Z , o u Z } o Ì & v P Ç } ( ^ Ç u Ç u Z o ] ( ] u µ v v o Ç À Ç ] v P Z Á Z ] Z ] À ] ] Z À ] } µ U }

The Boltzmann Machine

• The entire model can be viewed as a generative model• Has a probability of producing any binary vector :

Page 54: o E v } o Ì u v v D Z ] v · 2021. 3. 5. · d Z , o u Z } o Ì & v P Ç } ( ^ Ç u Ç u Z o ] ( ] u µ v v o Ç À Ç ] v P Z Á Z ] Z ] À ] ] Z À ] } µ U }

Training the network

• Training a Hopfield net: Must learn weights to “remember” target states and “dislike” other states– “State” == binary pattern of all the neurons

• Training Boltzmann machine: Must learn weights to assign a desired probability distribution to states – (vectors 𝐲, which we will now calls 𝑆 because I’m too lazy to normalize the notation)– This should assign more probability to patterns we “like” (or try to memorize) and less to

other patterns

Page 55: o E v } o Ì u v v D Z ] v · 2021. 3. 5. · d Z , o u Z } o Ì & v P Ç } ( ^ Ç u Ç u Z o ] ( ] u µ v v o Ç À Ç ] v P Z Á Z ] Z ] À ] ] Z À ] } µ U }

Training the network

• Must train the network to assign a desired probability distribution to states

• Given a set of “training” inputs – Assign higher probability to patterns seen more frequently– Assign lower probability to patterns that are not seen at all

• Alternately viewed: maximize likelihood of stored states

Visible Neurons

Page 56: o E v } o Ì u v v D Z ] v · 2021. 3. 5. · d Z , o u Z } o Ì & v P Ç } ( ^ Ç u Ç u Z o ] ( ] u µ v v o Ç À Ç ] v P Z Á Z ] Z ] À ] ] Z À ] } µ U }

Maximum Likelihood Training

• Maximize the average log likelihood of all “training” vectors – In the first summation, si and sj are bits of S– In the second, si’ and sj’ are bits of S’

∈𝐒

Average log likelihood of training vectors(to be maximized)

Page 57: o E v } o Ì u v v D Z ] v · 2021. 3. 5. · d Z , o u Z } o Ì & v P Ç } ( ^ Ç u Ç u Z o ] ( ] u µ v v o Ç À Ç ] v P Z Á Z ] Z ] À ] ] Z À ] } µ U }

Maximum Likelihood Training

• We will use gradient ascent, but we run into a problem..• The first term is just the average sisj over all training

patterns• But the second term is summed over all states

– Of which there can be an exponential number!

Page 58: o E v } o Ì u v v D Z ] v · 2021. 3. 5. · d Z , o u Z } o Ì & v P Ç } ( ^ Ç u Ç u Z o ] ( ] u µ v v o Ç À Ç ] v P Z Á Z ] Z ] À ] ] Z À ] } µ U }

The second term

• The second term is simply the expected value of sisj, over all possible values of the state

• We cannot compute it exhaustively, but we can compute it by sampling!

" ""

Page 59: o E v } o Ì u v v D Z ] v · 2021. 3. 5. · d Z , o u Z } o Ì & v P Ç } ( ^ Ç u Ç u Z o ] ( ] u µ v v o Ç À Ç ] v P Z Á Z ] Z ] À ] ] Z À ] } µ U }

Estimating the second term

• The expectation can be estimated as the average of samples drawn from the distribution

• Question: How do we draw samples from the Boltzmann distribution?– How do we draw samples from the network?

Page 60: o E v } o Ì u v v D Z ] v · 2021. 3. 5. · d Z , o u Z } o Ì & v P Ç } ( ^ Ç u Ç u Z o ] ( ] u µ v v o Ç À Ç ] v P Z Á Z ] Z ] À ] ] Z À ] } µ U }

The simulation solution

• Initialize the network randomly and let it “evolve”– By probabilistically selecting state values according to our model

• After many many epochs, take a snapshot of the state• Repeat this many many times• Let the collection of states be

, , ,

Page 61: o E v } o Ì u v v D Z ] v · 2021. 3. 5. · d Z , o u Z } o Ì & v P Ç } ( ^ Ç u Ç u Z o ] ( ] u µ v v o Ç À Ç ] v P Z Á Z ] Z ] À ] ] Z À ] } µ U }

The simulation solution for the second term

• The second term in the derivative is computed as the average of sampled states when the network is running “freely”

Page 62: o E v } o Ì u v v D Z ] v · 2021. 3. 5. · d Z , o u Z } o Ì & v P Ç } ( ^ Ç u Ç u Z o ] ( ] u µ v v o Ç À Ç ] v P Z Á Z ] Z ] À ] ] Z À ] } µ U }

Maximum Likelihood Training

• The overall gradient ascent rule

∈𝐒

∈𝐒

Sampled estimate

Page 63: o E v } o Ì u v v D Z ] v · 2021. 3. 5. · d Z , o u Z } o Ì & v P Ç } ( ^ Ç u Ç u Z o ] ( ] u µ v v o Ç À Ç ] v P Z Á Z ] Z ] À ] ] Z À ] } µ U }

Overall Training

• Initialize weights

• Let the network run to obtain simulated state samples

• Compute gradient and update weights

• Iterate

∈𝐒

Page 64: o E v } o Ì u v v D Z ] v · 2021. 3. 5. · d Z , o u Z } o Ì & v P Ç } ( ^ Ç u Ç u Z o ] ( ] u µ v v o Ç À Ç ] v P Z Á Z ] Z ] À ] ] Z À ] } µ U }

Overall Training

∈𝐒

state

Energy

Note the similarity to the update rule for the Hopfield network

Page 65: o E v } o Ì u v v D Z ] v · 2021. 3. 5. · d Z , o u Z } o Ì & v P Ç } ( ^ Ç u Ç u Z o ] ( ] u µ v v o Ç À Ç ] v P Z Á Z ] Z ] À ] ] Z À ] } µ U }

Adding Capacity to the Hopfield Network / Boltzmann Machine

• The network can store up to -bit patterns• How do we increase the capacity

65

Page 66: o E v } o Ì u v v D Z ] v · 2021. 3. 5. · d Z , o u Z } o Ì & v P Ç } ( ^ Ç u Ç u Z o ] ( ] u µ v v o Ç À Ç ] v P Z Á Z ] Z ] À ] ] Z À ] } µ U }

Expanding the network

• Add a large number of neurons whose actual values you don’t care about!

N Neurons K Neurons

66

Page 67: o E v } o Ì u v v D Z ] v · 2021. 3. 5. · d Z , o u Z } o Ì & v P Ç } ( ^ Ç u Ç u Z o ] ( ] u µ v v o Ç À Ç ] v P Z Á Z ] Z ] À ] ] Z À ] } µ U }

Expanded Network

• New capacity: patterns– Although we only care about the pattern of the first N

neurons– We’re interested in N-bit patterns

N Neurons K Neurons

67

Page 68: o E v } o Ì u v v D Z ] v · 2021. 3. 5. · d Z , o u Z } o Ì & v P Ç } ( ^ Ç u Ç u Z o ] ( ] u µ v v o Ç À Ç ] v P Z Á Z ] Z ] À ] ] Z À ] } µ U }

Terminology

• Terminology:– The neurons that store the actual patterns of interest: Visible

neurons– The neurons that only serve to increase the capacity but whose

actual values are not important: Hidden neurons– These can be set to anything in order to store a visible pattern

Visible Neurons

Hidden Neurons

Page 69: o E v } o Ì u v v D Z ] v · 2021. 3. 5. · d Z , o u Z } o Ì & v P Ç } ( ^ Ç u Ç u Z o ] ( ] u µ v v o Ç À Ç ] v P Z Á Z ] Z ] À ] ] Z À ] } µ U }

Training the network

• For a given pattern of visible neurons, there are any number of hidden patterns (2K)

• Which of these do we choose?– Ideally choose the one that results in the lowest energy

– But that’s an exponential search space!

Visible Neurons

Hidden Neurons

Page 70: o E v } o Ì u v v D Z ] v · 2021. 3. 5. · d Z , o u Z } o Ì & v P Ç } ( ^ Ç u Ç u Z o ] ( ] u µ v v o Ç À Ç ] v P Z Á Z ] Z ] À ] ] Z À ] } µ U }

The patterns• In fact we could have multiple hidden patterns

coupled with any visible pattern– These would be multiple stored patterns that all give

the same visible output– How many do we permit

• Do we need to specify one or more particular hidden patterns?– How about all of them– What do I mean by this bizarre statement?

Page 71: o E v } o Ì u v v D Z ] v · 2021. 3. 5. · d Z , o u Z } o Ì & v P Ç } ( ^ Ç u Ç u Z o ] ( ] u µ v v o Ç À Ç ] v P Z Á Z ] Z ] À ] ] Z À ] } µ U }

Boltzmann machine without hidden units

• This basic framework has no hidden units

• Extended to have hidden units

∈𝐒

Page 72: o E v } o Ì u v v D Z ] v · 2021. 3. 5. · d Z , o u Z } o Ì & v P Ç } ( ^ Ç u Ç u Z o ] ( ] u µ v v o Ç À Ç ] v P Z Á Z ] Z ] À ] ] Z À ] } µ U }

With hidden neurons

• Now, with hidden neurons the complete state pattern for even the training patterns is unknown– Since they are only defined over visible neurons

Visible Neurons

Hidden Neurons

Page 73: o E v } o Ì u v v D Z ] v · 2021. 3. 5. · d Z , o u Z } o Ì & v P Ç } ( ^ Ç u Ç u Z o ] ( ] u µ v v o Ç À Ç ] v P Z Á Z ] Z ] À ] ] Z À ] } µ U }

With hidden neurons

• We are interested in the marginal probabilities over visible bits– We want to learn to represent the visible bits– The hidden bits are the “latent” representation learned by the network

•

– = visible bits– = hidden bits

Visible Neurons

Hidden Neurons

Page 74: o E v } o Ì u v v D Z ] v · 2021. 3. 5. · d Z , o u Z } o Ì & v P Ç } ( ^ Ç u Ç u Z o ] ( ] u µ v v o Ç À Ç ] v P Z Á Z ] Z ] À ] ] Z À ] } µ U }

With hidden neurons

• We are interested in the marginal probabilities over visible bits– We want to learn to represent the visible bits– The hidden bits are the “latent” representation learned by the network

•

– = visible bits– = hidden bits

Visible Neurons

Hidden Neurons

Must train to maximize probability of desiredpatterns of visible bits

Page 75: o E v } o Ì u v v D Z ] v · 2021. 3. 5. · d Z , o u Z } o Ì & v P Ç } ( ^ Ç u Ç u Z o ] ( ] u µ v v o Ç À Ç ] v P Z Á Z ] Z ] À ] ] Z À ] } µ U }

Training the network

• Must train the network to assign a desired probability distribution to visible states

• Probability of visible state sums over all hidden states

Visible Neurons

Page 76: o E v } o Ì u v v D Z ] v · 2021. 3. 5. · d Z , o u Z } o Ì & v P Ç } ( ^ Ç u Ç u Z o ] ( ] u µ v v o Ç À Ç ] v P Z Á Z ] Z ] À ] ] Z À ] } µ U }

Maximum Likelihood Training

• Maximize the average log likelihood of all visible bits of “training” vectors 1 2 𝑁

– The first term also has the same format as the second term• Log of a sum

– Derivatives of the first term will have the same form as for the second term

∈𝐕

∈𝐕

Average log likelihood of training vectors(to be maximized)

Page 77: o E v } o Ì u v v D Z ] v · 2021. 3. 5. · d Z , o u Z } o Ì & v P Ç } ( ^ Ç u Ç u Z o ] ( ] u µ v v o Ç À Ç ] v P Z Á Z ] Z ] À ] ] Z À ] } µ U }

Maximum Likelihood Training

• We’ve derived this math earlier• But now both terms require summing over an exponential number of states

– The first term fixes visible bits, and sums over all configurations of hidden states for each visible configuration in our training set

– But the second term is summed over all states

∈𝐕

" "∈𝐕

" ""

∈𝐕

Page 78: o E v } o Ì u v v D Z ] v · 2021. 3. 5. · d Z , o u Z } o Ì & v P Ç } ( ^ Ç u Ç u Z o ] ( ] u µ v v o Ç À Ç ] v P Z Á Z ] Z ] À ] ] Z À ] } µ U }

The simulation solution

• The first term is computed as the average sampled hidden state with the visible bits fixed

• The second term in the derivative is computed as the average of sampled states when the network is running “freely”

∈𝐕

Page 79: o E v } o Ì u v v D Z ] v · 2021. 3. 5. · d Z , o u Z } o Ì & v P Ç } ( ^ Ç u Ç u Z o ] ( ] u µ v v o Ç À Ç ] v P Z Á Z ] Z ] À ] ] Z À ] } µ U }

More simulations

• Maximizing the marginal probability of requires summing over all values of – An exponential state space– So we will use simulations again

Visible Neurons

Hidden Neurons

Page 80: o E v } o Ì u v v D Z ] v · 2021. 3. 5. · d Z , o u Z } o Ì & v P Ç } ( ^ Ç u Ç u Z o ] ( ] u µ v v o Ç À Ç ] v P Z Á Z ] Z ] À ] ] Z À ] } µ U }

Step 1

• For each training pattern – Fix the visible units to – Let the hidden neurons evolve from a random initial point to

generate – Generate , ]

• Repeat K times to generate synthetic training

Visible Neurons

Hidden Neurons

Page 81: o E v } o Ì u v v D Z ] v · 2021. 3. 5. · d Z , o u Z } o Ì & v P Ç } ( ^ Ç u Ç u Z o ] ( ] u µ v v o Ç À Ç ] v P Z Á Z ] Z ] À ] ] Z À ] } µ U }

Step 2

• Now unclamp the visible units and let the entire network evolve several times to generate

Visible Neurons

Hidden Neurons

Page 82: o E v } o Ì u v v D Z ] v · 2021. 3. 5. · d Z , o u Z } o Ì & v P Ç } ( ^ Ç u Ç u Z o ] ( ] u µ v v o Ç À Ç ] v P Z Á Z ] Z ] À ] ] Z À ] } µ U }

Gradients

• Gradients are computed as before, except that the first term is now computed over the expanded training data

Page 83: o E v } o Ì u v v D Z ] v · 2021. 3. 5. · d Z , o u Z } o Ì & v P Ç } ( ^ Ç u Ç u Z o ] ( ] u µ v v o Ç À Ç ] v P Z Á Z ] Z ] À ] ] Z À ] } µ U }

Overall Training

• Initialize weights• Run simulations to get clamped and unclamped

training samples• Compute gradient and update weights• Iterate

𝑺 ∈𝐒

Page 84: o E v } o Ì u v v D Z ] v · 2021. 3. 5. · d Z , o u Z } o Ì & v P Ç } ( ^ Ç u Ç u Z o ] ( ] u µ v v o Ç À Ç ] v P Z Á Z ] Z ] À ] ] Z À ] } µ U }

Boltzmann machines

• Stochastic extension of Hopfield nets• Enables storage of many more patterns than

Hopfield nets• But also enables computation of probabilities

of patterns, and completion of pattern

Page 85: o E v } o Ì u v v D Z ] v · 2021. 3. 5. · d Z , o u Z } o Ì & v P Ç } ( ^ Ç u Ç u Z o ] ( ] u µ v v o Ç À Ç ] v P Z Á Z ] Z ] À ] ] Z À ] } µ U }

Boltzmann machines: Overall

• Training: Given a set of training patterns– Which could be repeated to represent relative probabilities

• Initialize weights• Run simulations to get clamped and unclamped training samples• Compute gradient and update weights• Iterate

𝑺 ∈𝐒

Page 86: o E v } o Ì u v v D Z ] v · 2021. 3. 5. · d Z , o u Z } o Ì & v P Ç } ( ^ Ç u Ç u Z o ] ( ] u µ v v o Ç À Ç ] v P Z Á Z ] Z ] À ] ] Z À ] } µ U }

Boltzmann machines: Overall

• Running: Pattern completion– “Anchor” the known visible units– Let the network evolve– Sample the unknown visible units

• Choose the most probable value

Page 87: o E v } o Ì u v v D Z ] v · 2021. 3. 5. · d Z , o u Z } o Ì & v P Ç } ( ^ Ç u Ç u Z o ] ( ] u µ v v o Ç À Ç ] v P Z Á Z ] Z ] À ] ] Z À ] } µ U }

Applications

• Filling out patterns• Denoising patterns• Computing conditional probabilities of patterns• Classification!!

– How?

Page 88: o E v } o Ì u v v D Z ] v · 2021. 3. 5. · d Z , o u Z } o Ì & v P Ç } ( ^ Ç u Ç u Z o ] ( ] u µ v v o Ç À Ç ] v P Z Á Z ] Z ] À ] ] Z À ] } µ U }

Boltzmann machines for classification

• Training patterns:– [f1, f2, f3, …. , class]– Features can have binarized or continuous valued representations– Classes have “one hot” representation

• Classification:– Given features, anchor features, estimate a posteriori probability

distribution over classes• Or choose most likely class

Page 89: o E v } o Ì u v v D Z ] v · 2021. 3. 5. · d Z , o u Z } o Ì & v P Ç } ( ^ Ç u Ç u Z o ] ( ] u µ v v o Ç À Ç ] v P Z Á Z ] Z ] À ] ] Z À ] } µ U }

Boltzmann machines: Issues

• Training takes for ever• Doesn’t really work for large problems

– A small number of training instances over a small number of bits

Page 90: o E v } o Ì u v v D Z ] v · 2021. 3. 5. · d Z , o u Z } o Ì & v P Ç } ( ^ Ç u Ç u Z o ] ( ] u µ v v o Ç À Ç ] v P Z Á Z ] Z ] À ] ] Z À ] } µ U }

Solution: Restricted Boltzmann Machines

• Partition visible and hidden units– Visible units ONLY talk to hidden units– Hidden units ONLY talk to visible units

• Restricted Boltzmann machine..– Originally proposed as “Harmonium Models” by Paul

Smolensky

VISIBLE

HIDDEN

Page 91: o E v } o Ì u v v D Z ] v · 2021. 3. 5. · d Z , o u Z } o Ì & v P Ç } ( ^ Ç u Ç u Z o ] ( ] u µ v v o Ç À Ç ] v P Z Á Z ] Z ] À ] ] Z À ] } µ U }

Solution: Restricted Boltzmann Machines

• Still obeys the same rules as a regular Boltzmann machine• But the modified structure adds a big benefit..

VISIBLE

HIDDEN

Page 92: o E v } o Ì u v v D Z ] v · 2021. 3. 5. · d Z , o u Z } o Ì & v P Ç } ( ^ Ç u Ç u Z o ] ( ] u µ v v o Ç À Ç ] v P Z Á Z ] Z ] À ] ] Z À ] } µ U }

Solution: Restricted Boltzmann Machines

VISIBLE

HIDDEN

VISIBLE

HIDDEN

Page 93: o E v } o Ì u v v D Z ] v · 2021. 3. 5. · d Z , o u Z } o Ì & v P Ç } ( ^ Ç u Ç u Z o ] ( ] u µ v v o Ç À Ç ] v P Z Á Z ] Z ] À ] ] Z À ] } µ U }

Recap: Training full Boltzmann machines: Step 1

• For each training pattern – Fix the visible units to – Let the hidden neurons evolve from a random initial point to

generate – Generate , ]

• Repeat K times to generate synthetic training

-1

1

1

1 -1

Visible Neurons Hidden Neurons

Page 94: o E v } o Ì u v v D Z ] v · 2021. 3. 5. · d Z , o u Z } o Ì & v P Ç } ( ^ Ç u Ç u Z o ] ( ] u µ v v o Ç À Ç ] v P Z Á Z ] Z ] À ] ] Z À ] } µ U }

Sampling: Restricted Boltzmann machine

• For each sample:– Anchor visible units– Sample from hidden units– No looping!!

VISIBLE

HIDDEN

Page 95: o E v } o Ì u v v D Z ] v · 2021. 3. 5. · d Z , o u Z } o Ì & v P Ç } ( ^ Ç u Ç u Z o ] ( ] u µ v v o Ç À Ç ] v P Z Á Z ] Z ] À ] ] Z À ] } µ U }

Recap: Training full Boltzmann machines: Step 2

• Now unclamp the visible units and let the entire network evolve several times to generate

-1

1

1

1 -1

Visible Neurons

Hidden Neurons

Page 96: o E v } o Ì u v v D Z ] v · 2021. 3. 5. · d Z , o u Z } o Ì & v P Ç } ( ^ Ç u Ç u Z o ] ( ] u µ v v o Ç À Ç ] v P Z Á Z ] Z ] À ] ] Z À ] } µ U }

Sampling: Restricted Boltzmann machine

• For each sample:– Iteratively sample hidden and visible units for a long time– Draw final sample of both hidden and visible units

VISIBLE

HIDDEN

Page 97: o E v } o Ì u v v D Z ] v · 2021. 3. 5. · d Z , o u Z } o Ì & v P Ç } ( ^ Ç u Ç u Z o ] ( ] u µ v v o Ç À Ç ] v P Z Á Z ] Z ] À ] ] Z À ] } µ U }

Pictorial representation of RBM training

• For each sample:– Initialize (visible) to training instance value– Iteratively generate hidden and visible units

• For a very long time

h0 h1 h2 h

v0 v1 v2 v

Page 98: o E v } o Ì u v v D Z ] v · 2021. 3. 5. · d Z , o u Z } o Ì & v P Ç } ( ^ Ç u Ç u Z o ] ( ] u µ v v o Ç À Ç ] v P Z Á Z ] Z ] À ] ] Z À ] } µ U }

Pictorial representation of RBM training

• Gradient (showing only one edge from visible node i to hidden node j)

• <vi, hj> represents average over many generated training samples

v0

h0

v1

h1

v2

h2

v

h

jiji

ijhvhv

w

vp 0)(log

i

j

i i i

j j j

Page 99: o E v } o Ì u v v D Z ] v · 2021. 3. 5. · d Z , o u Z } o Ì & v P Ç } ( ^ Ç u Ç u Z o ] ( ] u µ v v o Ç À Ç ] v P Z Á Z ] Z ] À ] ] Z À ] } µ U }

Recall: Hopfield Networks• Really no need to raise the entire surface, or even

every valley• Raise the neighborhood of each target memory

– Sufficient to make the memory a valley– The broader the neighborhood considered, the

broader the valley

99state

Energy

Page 100: o E v } o Ì u v v D Z ] v · 2021. 3. 5. · d Z , o u Z } o Ì & v P Ç } ( ^ Ç u Ç u Z o ] ( ] u µ v v o Ç À Ç ] v P Z Á Z ] Z ] À ] ] Z À ] } µ U }

A Shortcut: Contrastive Divergence

• Sufficient to run one iteration!

• This is sufficient to give you a good estimate of the gradient

v0

h0

v1

h1

10)(log

jijiij

hvhvw

vp

i

j

i

j

Page 101: o E v } o Ì u v v D Z ] v · 2021. 3. 5. · d Z , o u Z } o Ì & v P Ç } ( ^ Ç u Ç u Z o ] ( ] u µ v v o Ç À Ç ] v P Z Á Z ] Z ] À ] ] Z À ] } µ U }

Restricted Boltzmann Machines

• Excellent generative models for binary (or binarized) data

• Can also be extended to continuous-valued data– “Exponential Family Harmoniums with an Application

to Information Retrieval”, Welling et al., 2004

• Useful for classification and regression– How?– More commonly used to pretrain models

101

Page 102: o E v } o Ì u v v D Z ] v · 2021. 3. 5. · d Z , o u Z } o Ì & v P Ç } ( ^ Ç u Ç u Z o ] ( ] u µ v v o Ç À Ç ] v P Z Á Z ] Z ] À ] ] Z À ] } µ U }

Continuous-values RBMs

VISIBLE

HIDDEN

VISIBLE

HIDDEN

Hidden units may also be continuous values

Page 103: o E v } o Ì u v v D Z ] v · 2021. 3. 5. · d Z , o u Z } o Ì & v P Ç } ( ^ Ç u Ç u Z o ] ( ] u µ v v o Ç À Ç ] v P Z Á Z ] Z ] À ] ] Z À ] } µ U }

Other variants

• Left: “Deep” Boltzmann machines• Right: Helmholtz machine

– Trained by the “wake-sleep” algorithm

Page 104: o E v } o Ì u v v D Z ] v · 2021. 3. 5. · d Z , o u Z } o Ì & v P Ç } ( ^ Ç u Ç u Z o ] ( ] u µ v v o Ç À Ç ] v P Z Á Z ] Z ] À ] ] Z À ] } µ U }

Topics missed..

• Other algorithms for Learning and Inference over RBMs– Mean field approximations

• RBMs as feature extractors– Pre training

• RBMs as generative models• More structured DBMs• …

104


Rudolph the Red-Nosed Reindeer...& } Ç Z ^ v } Á u v U Á i } o o Ç Z Ç } µ o X t ] Z } v } ] v µ } v v } v Á } Ç u } µ } ( } o X & } Ç Z ^ v } Á u v U Á

Rudolph the Red-Nosed Reindeer...& } Ç Z ^ v } Á u v U Á i } o o Ç Z Ç } µ o X t ] Z } v } ] v µ } v v } v Á } Ç u } µ } ( } o X & } Ç Z ^ v } Á u v U Á

, } Á u v Ç u o } Ç } t Z v P } ( Z } u o } Ç v t ] Z } µ ] ] } v …...Z ] o Z o v ò õ ñ 9 õ ñ 9 ñ Á l u Ç Á ] v Z Z v P ] À v } Á í } P v ] Ì ] } v v } v r } (

, } Á u v Ç u o } Ç } t Z v P } ( Z } u o } Ç v t ] Z } µ ] ] } v …...Z ] o Z o v ò õ ñ 9 õ ñ 9 ñ Á l u Ç Á ] v Z Z v P ] À v } Á í } P v ] Ì ] } v v } v r } (

W Ç u } v ] } v W sKd z · 2013. 3. 25. · sKd z ^ ( } } v } ( v Ç } µ Ç v } v ] } v J / ( Ç } µ Z À ] P v Z v Á } v sKd z ^ ( } } v } ] u } À Ç } µ Ç v } v ] } v J hZ'

W Ç u } v ] } v W sKd z · 2013. 3. 25. · sKd z ^ ( } } v } ( v Ç } µ Ç v } v ] } v J / ( Ç } µ Z À ] P v Z v Á } v sKd z ^ ( } } v } ] u } À Ç } µ Ç v } v ] } v J hZ'

Æ ] u v ( } X d Z X í z W Z Ç ] > } } Ç 6LQJOH 6OLW ... Laboratory/… · Æ ] u v ( } x d z x í z w z Ç ] > } } Ç u v } ( w z Ç ] u //d z } } l î rwkhu kdqg vrxqg zdyhv

Æ ] u v ( } X d Z X í z W Z Ç ] > } } Ç 6LQJOH 6OLW ... Laboratory/… · Æ ] u v ( } x d z x í z w z Ç ] > } } Ç u v } ( w z Ç ] u //d z } } l î rwkhu kdqg vrxqg zdyhv

d Z Z } u v Ç [ î ì í ò ] À ] Ç Z } À } Z ] P Z o ] P Z Z ... · s o µ U } u } Ç Z Z } u v Ç U Á ] o o µ o ] v Z } ] ] À u } µ o ] v P } ( } Ç ...

d Z Z } u v Ç [ î ì í ò ] À ] Ç Z } À } Z ] P Z o ] P Z Z ... · s o µ U } u } Ç Z Z } u v Ç U Á ] o o µ o ] v Z } ] ] À u } µ o ] v P } ( } Ç ...

} Tc Ð T V e ¥ þ T V e ¥ r > I y q s Ò r > s Ò k z º ¥ { - s Q ^ Ç ç * Ò ¥ { - s k z ^ Ç ç - o 8 7 þ T V e ¥ Ç £ k z º Ç £ k z q V V ¥ T V e ¥ Ç £ q V V ¥

} Tc Ð T V e ¥ þ T V e ¥ r > I y q s Ò r > s Ò k z º ¥ { - s Q ^ Ç ç * Ò ¥ { - s k z ^ Ç ç - o 8 7 þ T V e ¥ Ç £ k z º Ç £ k z q V V ¥ T V e ¥ Ç £ q V V ¥

^ Ç v Z ] v ] } Ç v Z ] ^ µ ] } v Ç } u ] ] v - Thieme^ µ } Ç ^ Ç v Z ] v ] } Ç v Z ] ^ µ ] } v Ç } u ] ] v ^ µ ] ] Á X D Z U E ] Z } o Z X > U D Z Á : X Ç v U > ] r

^ Ç v Z ] v ] } Ç v Z ] ^ µ ] } v Ç } u ] ] v - Thieme^ µ } Ç ^ Ç v Z ] v ] } Ç v Z ] ^ µ ] } v Ç } u ] ] v ^ µ ] ] Á X D Z U E ] Z } o Z X > U D Z Á : X Ç v U > ] r

Yost - Whole Foods...d Z ] } µ u v ] } ] Ç v } v ( ] v ] o X E } } ( Z ] } µ u v u Ç ] o } ] v v Ç u v v } Z ] Ç Á ] Z } µ Z ] } Á ] v } v v } ( t Z } o & } } D l X

Yost - Whole Foods...d Z ] } µ u v ] } ] Ç v } v ( ] v ] o X E } } ( Z ] } µ u v u Ç ] o } ] v v Ç u v v } Z ] Ç Á ] Z } µ Z ] } Á ] v } v v } ( t Z } o & } } D l X

> v ] v P l t t l P ] v v ] v P ò Z : µ o Ç î ì î ì · 2020. 7. 1. · d Z K o Ç u ] Á u v } Z o Z ] Ç ] v d } l Ç } U ] v P } v Z î ð Z : µ o Ç v v ] v P } v Z õ

> v ] v P l t t l P ] v v ] v P ò Z : µ o Ç î ì î ì · 2020. 7. 1. · d Z K o Ç u ] Á u v } Z o Z ] Ç ] v d } l Ç } U ] v P } v Z î ð Z : µ o Ç v v ] v P } v Z õ

Brochure€¦ · /(Ç}µ vv} Z µ i Ç}µÁ] Z } µ Ç]v Z] o] U o } }v µ XdZ ] v Æ oo v Zv Z Á Á]oo o } vP }µ (} Ç}µ]v Ç}µ Z} v. o } ( µ ÇX u] W }P uu t Z}À Á v Ç ]+

Brochure€¦ · /(Ç}µ vv} Z µ i Ç}µÁ] Z } µ Ç]v Z] o] U o } }v µ XdZ ] v Æ oo v Zv Z Á Á]oo o } vP }µ (} Ç}µ]v Ç}µ Z} v. o } ( µ ÇX u] W }P uu t Z}À Á v Ç ]+

K À } & o ] Ç Z À ] Á v v µ o Z } µ Ç Z } P } µ v Ç U î ì î ì

K À } & o ] Ç Z À ] Á v v µ o Z } µ Ç Z } P } µ v Ç U î ì î ì

] v µ P Z Z ] o v [ , } ] o Z ] Ç d µ [ Z } v } µ v z ... · Z ] o v v Ç } µ v P } o [ o ] À o ] v µ Ç ] o o v X

] v µ P Z Z ] o v [ , } ] o Z ] Ç d µ [ Z } v } µ v z ... · Z ] o v v Ç } µ v P } o [ o ] À o ] v µ Ç ] o o v X

}v W }µv Ç }} ]v } W · ] Z ( v ^Ç u(} Z v o ]v,]PZ µ }v ?:5*%)/%)*@ /v u } P]}v }( Z Á} o } Ç hv]À ] v D] v] ] ZÀ P }v ] Ç u

}v W }µv Ç }} ]v } W · ] Z ( v ^Ç u(} Z v o ]v,]PZ µ }v ?:5*%)/%)*@ /v u } P]}v }( Z Á} o } Ç hv]À ] v D] v] ] ZÀ P }v ] Ç u

W d^ } } o } ] u } À } µ v ] o ] Ç v v v Ç ] v µ o ] À ] · u ] v v v v } ] } v v ( } µ v Z } v o Ç ó ó 9 } ( } o o o } ] } v ( } o u v Ç v Z ] P Z Z } } o

W d^ } } o } ] u } À } µ v ] o ] Ç v v v Ç ] v µ o ] À ] · u ] v v v v } ] } v v ( } µ v Z } v o Ç ó ó 9 } ( } o o o } ] } v ( } o u v Ç v Z ] P Z Z } } o

*ROGPDQ ,V * : } µ v Ç } ( Z ^ } µ o X...i } ] v ( } Á ] Z ^ v X t Z Ç } , o o } Á o o Z ] o u ] ( } µ v } ( o o µ M v Ç U Á l v } Á Z Z o o Ç v µ o Ç } o } À µ X

*ROGPDQ ,V * : } µ v Ç } ( Z ^ } µ o X...i } ] v ( } Á ] Z ^ v X t Z Ç } , o o } Á o o Z ] o u ] ( } µ v } ( o o µ M v Ç U Á l v } Á Z Z o o Ç v µ o Ç } o } À µ X

t Z P Ç Á Z ( } P } o ( } v & ] Ç U } o Ç } v } ( Z Ç } o ...

t Z P Ç Á Z ( } P } o ( } v & ] Ç U } o Ç } v } ( Z Ç } o ...

} Ç } ( Z ' v >

} Ç } ( Z ' v >

, À Ç } µ ~ } Z Ç ] Ç } µ ~ } Æ ] v , Z Z v P } Z , Z v ......, o Z } Ç } µ ( ] o o ] v P Z ] µ À Ç ] v } µ , o Z } Ç } µ ( ] o o ] v P Z ] µ À Ç ] v } µ

, À Ç } µ ~ } Z Ç ] Ç } µ ~ } Æ ] v , Z Z v P } Z , Z v ......, o Z } Ç } µ ( ] o o ] v P Z ] µ À Ç ] v } µ , o Z } Ç } µ ( ] o o ] v P Z ] µ À Ç ] v } µ

d Z ] v } À ] v Z v Ç l

d Z ] v } À ] v Z v Ç l

Á ( ] o ] Ç Z }€¦ · î ì î ì D ' Z t Á ( ] o ] Ç Z } ð ì õ> Ç v v l ] v Ç Z o ] D ^ o ] l t P

Á ( ] o ] Ç Z }€¦ · î ì î ì D ' Z t Á ( ] o ] Ç Z } ð ì õ> Ç v v l ] v Ç Z o ] D ^ o ] l t P