Post on 19-Aug-2020
Unsupervised learning with Artificial Neural Networks
Guénaël Cabanes
Introduction
2019 Guénaël Cabanes 2
Unsupervised learning
Unsupervised learning try to estimate a model of the data underlying distribution
2019 Guénaël Cabanes 3
Unsupervised learning
2019 Guénaël Cabanes 4
Feedforward Networks
2019 Guénaël Cabanes 5
Unsupervised Deep Learning
2019 Guénaël Cabanes 6
Generative Adversarial Networks (GAN)
2019 Guénaël Cabanes 7
Generative• Learn a generative model
Adversarial• Trained in an adversarial setting
Networks• Use Deep Neural Networks
GAN: why generative?
2019 Guénaël Cabanes 8
We’ve seen discriminative models • Given an image X, predict a label Y• Estimates P(Y|X)
Discriminative models have several key limitations• Can’t model P(X), i.e. the probability of seeing a certain image• Thus, can’t sample from P(X), e.g. can’t generate new data
Generative models (in general) cope with all of above• Can model P(X)• Can generate new data
GAN: which face is fake?
2019 Guénaël Cabanes 9
GAN: architecture
2019 Guénaël Cabanes 10
• Z is some random noise (Gaussian/Uniform).• Z can be thought as the latent representation of the image.
https://www.slideshare.net/xavigiro/
GAN: architecture
2019 Guénaël Cabanes 11
GAN: architecture
2019 Guénaël Cabanes 12
GAN: architecture
2019 Guénaël Cabanes 13
GAN: architecture
2019 Guénaël Cabanes 14
GAN: intuition
2019 Guénaël Cabanes 15
GAN: intuition
2019 Guénaël Cabanes 16
GAN: intuition
2019 Guénaël Cabanes 17
GAN: intuition
2019 Guénaël Cabanes 18
GAN: intuition
2019 Guénaël Cabanes 19
GAN: intuition
2019 Guénaël Cabanes 20
GAN: intuition
2019 Guénaël Cabanes 21
GAN: intuition
2019 Guénaël Cabanes 22
GAN: intuition
2019 Guénaël Cabanes 23
GAN: intuition
2019 Guénaël Cabanes 24
GAN: intuition
2019 Guénaël Cabanes 25
GAN: intuition
2019 Guénaël Cabanes 26
GAN: intuition
2019 Guénaël Cabanes 27
GAN: intuition
2019 Guénaël Cabanes 28
GAN: intuition
2019 Guénaël Cabanes 29
GAN: intuition
2019 Guénaël Cabanes 30
GAN: intuition
2019 Guénaël Cabanes 31
GAN: face generation
2019 Guénaël Cabanes 32
GAN: face generation
2019 Guénaël Cabanes 33
GAN: face generation
2019 Guénaël Cabanes 34
GAN: Convolutional Network
2019 Guénaël Cabanes 35
ReLU = rectified linear unit
GAN: Convolutional Network
2019 Guénaël Cabanes 36
Feature hierarchies
2019 Guénaël Cabanes 37
pixels
edges
object parts(combination of edges)
object models
GAN: Convolutional Network
Convolutional Neural Network
2019 Guénaël Cabanes 38
Convolutional Neural Network
2019 Guénaël Cabanes 39
GAN: Convolutional Network
2019 Guénaël Cabanes 40
GAN: Convolutional Network
2019 Guénaël Cabanes 41
GAN: Convolutional Network
2019 Guénaël Cabanes 42
GAN: Convolutional Network
2019 Guénaël Cabanes 43
GAN: Convolutional Network
2019 Guénaël Cabanes 44
GAN: Convolutional Network
2019 Guénaël Cabanes 45
GAN: Convolutional Network
2019 Guénaël Cabanes 46
GAN: Convolutional Network
2019 Guénaël Cabanes 47
GAN: Convolutional Network
2019 Guénaël Cabanes 48
GAN: Convolutional Network
2019 Guénaël Cabanes 49
GAN: Convolutional Network
2019 Guénaël Cabanes 50
GAN: Convolutional Network
2019 Guénaël Cabanes 51
GAN: Convolutional Network
2019 Guénaël Cabanes 52
GAN: Convolutional Network
2019 Guénaël Cabanes 53
GAN: Convolutional Network
2019 Guénaël Cabanes 54
GAN: Convolutional Network
2019 Guénaël Cabanes 55
GAN: Pooling
2019 Guénaël Cabanes 56
GAN: Pooling
2019 Guénaël Cabanes 57
3x3 pooling over 5x5 convolved feature
GAN: Convolutional Architecture
2019 Guénaël Cabanes 58
GAN: Convolutional Architecture
2019 Guénaël Cabanes 59
GAN: Convolutional Architecture
2019 Guénaël Cabanes 60
GAN: Convolutional Architecture
2019 Guénaël Cabanes 61
GAN: face generation
2019 Guénaël Cabanes 62
GAN: anime face generation
2019 Guénaël Cabanes 63
GAN: interpolation
2019 Guénaël Cabanes 64
https://arxiv.org/pdf/1703.10717.pdf
GAN: face aging
2019 Guénaël Cabanes 65
GAN: adding attributes
2019 Guénaël Cabanes 66
GAN: combination
2019 Guénaël Cabanes 67
GAN: image reconstruction
2019 Guénaël Cabanes 68
GAN: photo to paint
2019 Guénaël Cabanes 69
GAN: artistic creation
2019 Guénaël Cabanes 70
http://people.eecs.berkeley.edu/~junyanz/projects/gvm/
GAN: text to image
2019 Guénaël Cabanes 71
GAN: text to image
2019 Guénaël Cabanes 72
• Sample in parallel
• Few restrictions on generator function
• Subjectively better results
• Very difficult to train properly
• Difficult to evaluate
• Likelihood cannot be computed
• No encoder (in vanilla GAN)
2014 Guénaël Cabanes 73
GAN: advantages/drawbacks
Topological Networks
2019 Guénaël Cabanes 74
75
K-means (reminders)
Cost function :
2019 Guénaël Cabanes
Self-Organizing Maps (SOM)
2019 Guénaël Cabanes 76
SOM: Principles
2019 Guénaël Cabanes 77
SOM: Cost function
2019 Guénaël Cabanes 78
SOM: Topology
2019 Guénaël Cabanes 79
SOM: Learning
2019 Guénaël Cabanes 80
SOM: Learning
2019 Guénaël Cabanes 81
SOM: Algorithm
2019 Guénaël Cabanes 82
SOM: Algorithm
2019 Guénaël Cabanes 83
Cost function to minimize
Gradient
Adaptation rules
Affectation
Neighborhood function
Distance on the map
Temperature
SOM: Algorithm
2019 Guénaël Cabanes 84
position of neuron s
SOM: Examples
2019 Guénaël Cabanes 85
SOM: Examples
2019 Guénaël Cabanes 86
SOM: Examples
2019 Guénaël Cabanes 87
SOM: Application 1
2019 Guénaël Cabanes 88
SOM: Application 1
2019 Guénaël Cabanes 89
SOM: Interpretation
2019 Guénaël Cabanes 90
SOM: Interpretation
2019 Guénaël Cabanes 91
SOM: Application 2
2019 Guénaël Cabanes 92
World Bank poverty indicators (1992)
39 variables
SOM: Application 2
2019 Guénaël Cabanes 93
SOM: Application 2
2019 Guénaël Cabanes 94
Deep SOM
2019 Guénaël Cabanes 95
Deep SOM
2019 Guénaël Cabanes 96
Euclidean Distance
Kullback–Leibler divergence
Prototypes are distributions
Prototypes are vectors
Properties
2019 Guénaël Cabanes 97
Erro
r
Properties
2019 Guénaël Cabanes 98
1 layer 5 layers
• Model is easily interpreted and understood.
• Provide intelligible visualisation of the data.
• Clustering large and complex data sets.
• Lots of parameters to choose.
• Can generate topological anomalies.
• Not very fast
2014 Guénaël Cabanes 99
SOM: advantages/drawbacks
Growing Neural Gas (GNG)
2019 Guénaël Cabanes 100
GNG: Algorithm
2019 Guénaël Cabanes 101
• With s and t the two best neurons for a data point x:– Updating errors and utilities:
• Updating ages:– Increment by 1 the age of all edges between s and its topological neighbours. – If needed, create an edge between s and t, then set the age of this edge to 0.
• At the end of each step, for each neuron:
GNG: Algorithm
2019 Guénaël Cabanes 102
Updating prototypes:
GNG: Algorithm
2019 Guénaël Cabanes 103
Adding a new neuron:
GNG: Algorithm
2019 Guénaël Cabanes 104
Removing a neuron:
GNG: Algorithm
2019 Guénaël Cabanes 105
Removing an edge:
GNG: Example
2019 Guénaël Cabanes 106
Density-based – GNG
2019 Guénaël Cabanes 107
2019 Guénaël Cabanes 108
Density-based – GNG
2019 Guénaël Cabanes 109
Density-based – GNG
2019 Guénaël Cabanes 110
Density-based – GNG
2019 Guénaël Cabanes 111
Density-based – GNG
2019 Guénaël Cabanes 112
Density-based – GNG
2019 Guénaël Cabanes 113
Density-based – GNG
GNG vs SOM
2019 Guénaël Cabanes 114
• Learn the topology.
• Is adapted to data streams.
• Clustering large and complex data sets.
• Lots of parameters to choose.
• No simple visualization.
• Not very fast
2014 Guénaël Cabanes 115
GNG: advantages/drawbacks
Summary
2019 116Guénaël Cabanes
You need useful representations and visualizations of the data structure:
➢ Self Organizing Map
You need a synthetic description of a dynamic data structure:
➢ Growing Neural Gas
You need to generate new data:
➢ Generative Adversarial Networks
Sources
• http://eric.univ-lyon2.fr/~ricco/cours/slides/en/kohonen_som.pdf
• http://www.cis.hut.fi/research/som-research/worldmap.html
• https://www.demogng.de/JavaPaper/node19.html
• https://www.slideshare.net/zukun/p05-deep-boltzmann-machines-cvpr2012-deep-learning-methods-for-vision
• https://skymind.ai/wiki/generative-adversarial-network-gan
• https://medium.com/@jonathan_hui/gan-some-cool-applications-of-gans-4c9ecca35900
• https://medium.com/datadriveninvestor/text-to-image-synthesis-6e5de1bf86ec
2019 Guénaël Cabanes 117