Survey

Deep Learning Survey

I.I. Itauma

Wayne State UniversityDepartment of Computer Science

February 15, 2013

Itauma Deep Learning

Survey Learning

What is Deep Learning?

Yoshua Bengio ACL201 [2] "Deep Learningalgorithms attempt to learn multiple levels ofrepresentation of increasing complexity/abstraction".

Most current Machine Learning works well becauseof human designed representations and inputfeafures.

ML becomes just optimizing weights to best make afinal prediction.

Represenation learning attempts to automaticallylearn good features or represenation.

Deep Learning is a new area of ML research, whichhas been introduced with the objective of moving MLcloser to one of its original goals ArtificialIntelligence.

Itauma Deep Learning

Survey Learning

Breakthrough in Learning Deep Architectures

Before 2006, training deep architectures wasunsuccessful.

Hinton’s [1] and Yoshua Bengio [2] et al discovered:

Unsupervised learning of representations can beused to (pre-)train each layer.

Unsupervised training of one layer at a time, on topof the previously trained ones. The representationlearned at each level is the input for the next layer.

Using supervised training to fine-tune all the layers(in addition to one or more additional layers that arededicated to producing predictions).

Itauma Deep Learning

Survey Learning

Applications of Deep Learning

DL is about learning representation features. Handcraftingfeatures is time-consuming.

The feasures are often both over-specified and incomplete.

DL provides a way of developing representation forlearning and reasoning like humans.

DL has been used successfully in Speech recognition,NLP and visual perception.

HDFS exposes block placement so that computation canbe migrated to data

Itauma Deep Learning

Survey Learning

Visual perception with Deep Learning

Yann Le Cun (Google Tech Talks 2008) [4] investigatedwhy we learn perception.

He defined Deep Learning as learning a hierarchy ofinternal representation.

From low-level features to mid-level invariantrepresentation to object identities.

Itauma Deep Learning

Survey Learning

Unsupervised Learning Feature as Density Estimation.

We need unsupervised learning methods that can learninvariant feature hierarchies.

Itauma Deep Learning

Survey Learning

Unsupervised Learning

Probabilistic View:Produces a probability density function that:has high value in regions of high sample density.has low value everywhere else.

Energy-based View:Produces a enery function E(Y,W) that:has low value in regions of high sample density.has high value everywhere else.

Itauma Deep Learning

Survey Learning

Deep Learning Tutorial

The tutorials presented [5] introduce some of the mostimportant deep learning algorithms.It also shows how to run them using Theano.Theano is a python library that makes writing DL modelseasy, and gives the option of training them on a GPU.The following topics are discussed:

Logistic Regression - using Theano for something simple.Multilayer perceptron - introduction to layers.Deep Convolutional Network - a simple version of LeNet5.Auto Encoders, Denoising Autoencoders - description ofautoencoders.Stacked Denoising Auto-Encoders - easy steps intounsupervised pre-training for deep nets.Restricted Boltzmann Machines - single layer generativeRBM model.Deep Belief Networks - unsupervised generativepre-training of stacked RBMs followed by supervisedfine-tuning.

Itauma Deep Learning

Survey Learning

Conclusion

We need unsupervised Learning methods that can learninvariant feature hierarchies.

In Hinton [1] paper, the DBN uses RBM for unsupervisedlearning of representation of each layer.

Yann [4] presented methods to learn hierarchies of sparseand invariant features.

Deep Learning performs better than most traditional"shallow" architectures like SVM for recognition.

In traditional architectures, the trainable classifier is oftengeneric (task independent).

Itauma Deep Learning

Appendix References

References I

Hinton, G. E., Osindero, S. and Teh, Y., A fast learningalgorithm for deep belief nets Neural Computation18:1527-1554, 2006

Yoshua Bengio, Pascal Lamblin, Dan Popovici and HugoLarochelle, Greedy Layer-Wise Training of Deep Networks,in J. Platt et al. (Eds), Advances in Neural InformationProcessing Systems 19 (NIPS 2006), pp. 153-160, MITPress, 2007

Marc’Aurelio Ranzato, Christopher Poultney, Sumit Chopraand Yann LeCun Efficient Learning of SparseRepresentations with an Energy-Based Model, in J. Platt etal. (Eds), Advances in Neural Information ProcessingSystems (NIPS 2006), MIT Press, 2007

Itauma Deep Learning

Appendix References

References II

Visual Perception with Deep Learninghttp://www.youtube.com/watch?v=3boKlkPBckA

Deep Learning Tutorialhttp://deeplearning.net/tutorial/

Itauma Deep Learning

Appendix References

Thanks!

Itauma Deep Learning