Day 2 Lecture 1
Deep Belief Networks (DBN)
Elisa Sayrol
Restrictive Boltzmann Machine (RBM)
Training. Contrastive Divergence (CD)
Deep Belief Networks (DBN)
Overview
3
Restricted Boltzmann Machine (RBM)
Figure: Geoffrey Hinton (2013)
Salakhutdinov, Ruslan, Andriy Mnih, and Geoffrey Hinton. "Restricted Boltzmann machines for collaborative filtering." Proceedings of the 24th international conference on Machine learning. ACM, 2007.
● Shallow two-layer net.● Restricted=No two nodes in a layer share a
connection● Bipartite graph.● Bidirectional graph
○ Shared weights.○ Different biases.
4
Restricted Boltzmann Machine (RBM)
Figure: Geoffrey Hinton (2013)
Salakhutdinov, Ruslan, Andriy Mnih, and Geoffrey Hinton. "Restricted Boltzmann machines for collaborative filtering." Proceedings of the 24th international conference on Machine learning. ACM, 2007.
5
Restricted Boltzmann Machine (RBM)
DeepLearning4j, “A Beginner’s Tutorial for Restricted Boltzmann Machines”.
Forward pass
6
Restricted Boltzmann Machine (RBM)
DeepLearning4j, “A Beginner’s Tutorial for Restricted Boltzmann Machines”.
Backwardpass
c
c
c
c
7
Restricted Boltzmann Machine (RBM)
DeepLearning4j, “A Beginner’s Tutorial for Restricted Boltzmann Machines”.
Backwardpass
The reconstructed values at the visible layer are compared with the actual ones with the KL Divergence.
8
What are the Maths behind RBMs? (Estimation of the parameters)
Geoffrey Hinton, "Introduction to Deep Learning & Deep Belief Nets” (2012)Geoorey Hinton, “Tutorial on Deep Belief Networks”. NIPS 2007.
9
What are the Maths behind RBMs?
Other references:
Deeplearning.net: Restricted Boltzmann Machines (with Theano functions and concepts)Hugo Larochelle: Course on NN
Let’s take a look at some of his slides on RBM….
19
Deep Belief Networks (DBN)
Hinton, Geoffrey E., Simon Osindero, and Yee-Whye Teh. "A fast learning algorithm for deep belief nets." Neural computation 18, no. 7 (2006): 1527-1554.
● Architecture like an MLP.● Training as a stack of
RBMs.
20
Deep Belief Networks (DBN)
Hinton, Geoffrey E., Simon Osindero, and Yee-Whye Teh. "A fast learning algorithm for deep belief nets." Neural computation 18, no. 7 (2006): 1527-1554.
● Architecture like an MLP.● Training as a stack of
RBMs.
21
Deep Belief Networks (DBN)
Hinton, Geoffrey E., Simon Osindero, and Yee-Whye Teh. "A fast learning algorithm for deep belief nets." Neural computation 18, no. 7 (2006): 1527-1554.
● Architecture like an MLP.● Training as a stack of
RBMs.
22
Deep Belief Networks (DBN)
Hinton, Geoffrey E., Simon Osindero, and Yee-Whye Teh. "A fast learning algorithm for deep belief nets." Neural computation 18, no. 7 (2006): 1527-1554.
● Architecture like an MLP.● Training as a stack of
RBMs.
23
Deep Belief Networks (DBN)
Hinton, Geoffrey E., Simon Osindero, and Yee-Whye Teh. "A fast learning algorithm for deep belief nets." Neural computation 18, no. 7 (2006): 1527-1554.
● Architecture like an MLP.● Training as a stack of
RBMs…● ...so they do not need
labels:
Unsupervisedlearning
24
Deep Belief Networks (DBN)
Hinton, Geoffrey E., Simon Osindero, and Yee-Whye Teh. "A fast learning algorithm for deep belief nets." Neural computation 18, no. 7 (2006): 1527-1554.
After the DBN is trained, it can be fine-tuned with a reduced amount of labels to solve a supervised task with superior performance.
Supervisedlearning
Softm
ax
Top Related