Unsupervised learning with Artificial Neural Networksbennani/tmpc/RNA/RNA-6.pdfwith Artificial...

Unsupervised learning with Artificial Neural Networks

Guénaël Cabanes

Introduction

2019 Guénaël Cabanes 2

Unsupervised learning

Unsupervised learning try to estimate a model of the data underlying distribution


Unsupervised learning


Feedforward Networks


Unsupervised Deep Learning


Generative Adversarial Networks (GAN)


Generative• Learn a generative model

Adversarial• Trained in an adversarial setting

Networks• Use Deep Neural Networks

GAN: why generative?


We’ve seen discriminative models • Given an image X, predict a label Y• Estimates P(Y|X)

Discriminative models have several key limitations• Can’t model P(X), i.e. the probability of seeing a certain image• Thus, can’t sample from P(X), e.g. can’t generate new data

Generative models (in general) cope with all of above• Can model P(X)• Can generate new data

GAN: which face is fake?


GAN: architecture


• Z is some random noise (Gaussian/Uniform).• Z can be thought as the latent representation of the image.

https://www.slideshare.net/xavigiro/

GAN: architecture


GAN: intuition


GAN: face generation


GAN: Convolutional Network


ReLU = rectified linear unit

Feature hierarchies


pixels

edges

object parts(combination of edges)

object models


Convolutional Neural Network


GAN: Pooling


GAN: Pooling


3x3 pooling over 5x5 convolved feature

GAN: Convolutional Architecture


GAN: anime face generation


GAN: interpolation


https://arxiv.org/pdf/1703.10717.pdf

GAN: face aging


GAN: adding attributes


GAN: combination


GAN: image reconstruction


GAN: photo to paint


GAN: artistic creation


http://people.eecs.berkeley.edu/~junyanz/projects/gvm/

GAN: text to image


• Sample in parallel

• Few restrictions on generator function

• Subjectively better results

• Very difficult to train properly

• Difficult to evaluate

• Likelihood cannot be computed

• No encoder (in vanilla GAN)


GAN: advantages/drawbacks

Topological Networks


75

K-means (reminders)

Cost function :

2019 Guénaël Cabanes

Self-Organizing Maps (SOM)


SOM: Principles


SOM: Cost function


SOM: Topology


SOM: Learning


SOM: Algorithm


SOM: Algorithm


Cost function to minimize

Gradient

Adaptation rules

Affectation

Neighborhood function

Distance on the map

Temperature

SOM: Algorithm


position of neuron s

SOM: Examples


SOM: Application 1


SOM: Interpretation


SOM: Application 2


World Bank poverty indicators (1992)

39 variables

SOM: Application 2


Deep SOM


Deep SOM


Euclidean Distance

Kullback–Leibler divergence

Prototypes are distributions

Prototypes are vectors

Properties


Erro

r

Properties


1 layer 5 layers

• Model is easily interpreted and understood.

• Provide intelligible visualisation of the data.

• Clustering large and complex data sets.

• Lots of parameters to choose.

• Can generate topological anomalies.

• Not very fast


SOM: advantages/drawbacks

Growing Neural Gas (GNG)


GNG: Algorithm


• With s and t the two best neurons for a data point x:– Updating errors and utilities:

• Updating ages:– Increment by 1 the age of all edges between s and its topological neighbours. – If needed, create an edge between s and t, then set the age of this edge to 0.

• At the end of each step, for each neuron:

GNG: Algorithm


Updating prototypes:

GNG: Algorithm


Adding a new neuron:

GNG: Algorithm


Removing a neuron:

GNG: Algorithm


Removing an edge:

GNG: Example


Density-based – GNG


GNG vs SOM


• Learn the topology.

• Is adapted to data streams.

• Clustering large and complex data sets.

• Lots of parameters to choose.

• No simple visualization.

• Not very fast


GNG: advantages/drawbacks

Summary

2019 116Guénaël Cabanes

You need useful representations and visualizations of the data structure:

➢ Self Organizing Map

You need a synthetic description of a dynamic data structure:

➢ Growing Neural Gas

You need to generate new data:

➢ Generative Adversarial Networks

Sources

• http://eric.univ-lyon2.fr/~ricco/cours/slides/en/kohonen_som.pdf

• http://www.cis.hut.fi/research/som-research/worldmap.html

• https://www.demogng.de/JavaPaper/node19.html

• https://www.slideshare.net/zukun/p05-deep-boltzmann-machines-cvpr2012-deep-learning-methods-for-vision

• https://skymind.ai/wiki/generative-adversarial-network-gan

• https://medium.com/@jonathan_hui/gan-some-cool-applications-of-gans-4c9ecca35900

• https://medium.com/datadriveninvestor/text-to-image-synthesis-6e5de1bf86ec


Unsupervised learning with Artificial Neural Networksbennani/tmpc/RNA/RNA-6.pdfwith Artificial...

Documents

Transcript of Unsupervised learning with Artificial Neural Networksbennani/tmpc/RNA/RNA-6.pdfwith Artificial...