Introduction to Generative Models (and GANs) · 2020-02-06 · Introduction to Generative Models...
Transcript of Introduction to Generative Models (and GANs) · 2020-02-06 · Introduction to Generative Models...
![Page 1: Introduction to Generative Models (and GANs) · 2020-02-06 · Introduction to Generative Models (and GANs) Haoqiang Fan fhq@megvii.com April. 2019 Figures adapted from NIPS 2016](https://reader031.fdocuments.net/reader031/viewer/2022041017/5ecaa2dfe85e912d244177dd/html5/thumbnails/1.jpg)
Introduction to Generative
Models (and GANs)Haoqiang Fan
April. 2019
Figures adapted from NIPS 2016 Tutorial Generative Adversarial Netw orks
![Page 2: Introduction to Generative Models (and GANs) · 2020-02-06 · Introduction to Generative Models (and GANs) Haoqiang Fan fhq@megvii.com April. 2019 Figures adapted from NIPS 2016](https://reader031.fdocuments.net/reader031/viewer/2022041017/5ecaa2dfe85e912d244177dd/html5/thumbnails/2.jpg)
Agenda Today
● GAN
● 3D Reconstruction
● Misc topics in Computational Photography
![Page 3: Introduction to Generative Models (and GANs) · 2020-02-06 · Introduction to Generative Models (and GANs) Haoqiang Fan fhq@megvii.com April. 2019 Figures adapted from NIPS 2016](https://reader031.fdocuments.net/reader031/viewer/2022041017/5ecaa2dfe85e912d244177dd/html5/thumbnails/3.jpg)
Generative Models: Learning the Distributions
Discriminative: learns the likelihood
Generative: performs Density Estimation (learns the distribution) to allow sampling
Figures adapted from NIPS 2016 Tutorial Generative Adversarial Netw orks
![Page 4: Introduction to Generative Models (and GANs) · 2020-02-06 · Introduction to Generative Models (and GANs) Haoqiang Fan fhq@megvii.com April. 2019 Figures adapted from NIPS 2016](https://reader031.fdocuments.net/reader031/viewer/2022041017/5ecaa2dfe85e912d244177dd/html5/thumbnails/4.jpg)
Loss function for distribution: Ambiguity and the
“blur” effect
Figures adapted from NIPS 2016 Tutorial Generative Adversarial Netw orks
GenerativeMSE: a Discriminative model just smoothes all possibilities.
![Page 5: Introduction to Generative Models (and GANs) · 2020-02-06 · Introduction to Generative Models (and GANs) Haoqiang Fan fhq@megvii.com April. 2019 Figures adapted from NIPS 2016](https://reader031.fdocuments.net/reader031/viewer/2022041017/5ecaa2dfe85e912d244177dd/html5/thumbnails/5.jpg)
Ambiguity and the “blur” effect
Figures adapted from NIPS 2016 Tutorial Generative Adversarial Netw orks
Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network
![Page 6: Introduction to Generative Models (and GANs) · 2020-02-06 · Introduction to Generative Models (and GANs) Haoqiang Fan fhq@megvii.com April. 2019 Figures adapted from NIPS 2016](https://reader031.fdocuments.net/reader031/viewer/2022041017/5ecaa2dfe85e912d244177dd/html5/thumbnails/6.jpg)
Example Application of Generative
Models
![Page 7: Introduction to Generative Models (and GANs) · 2020-02-06 · Introduction to Generative Models (and GANs) Haoqiang Fan fhq@megvii.com April. 2019 Figures adapted from NIPS 2016](https://reader031.fdocuments.net/reader031/viewer/2022041017/5ecaa2dfe85e912d244177dd/html5/thumbnails/7.jpg)
Image Generation from Sketch
Figures adapted from NIPS 2016 Tutorial Generative Adversarial Netw orks
iGAN: Interactive Image Generation via Generative Adversarial Networks
![Page 8: Introduction to Generative Models (and GANs) · 2020-02-06 · Introduction to Generative Models (and GANs) Haoqiang Fan fhq@megvii.com April. 2019 Figures adapted from NIPS 2016](https://reader031.fdocuments.net/reader031/viewer/2022041017/5ecaa2dfe85e912d244177dd/html5/thumbnails/8.jpg)
Interactive Editing
Figures adapted from NIPS 2016 Tutorial Generative Adversarial Netw orks
Neural Photo Editing with Introspective Adversarial Networks
![Page 9: Introduction to Generative Models (and GANs) · 2020-02-06 · Introduction to Generative Models (and GANs) Haoqiang Fan fhq@megvii.com April. 2019 Figures adapted from NIPS 2016](https://reader031.fdocuments.net/reader031/viewer/2022041017/5ecaa2dfe85e912d244177dd/html5/thumbnails/9.jpg)
Image to Image Translation
Figures adapted from NIPS 2016 Tutorial Generative Adversarial Netw orks
![Page 10: Introduction to Generative Models (and GANs) · 2020-02-06 · Introduction to Generative Models (and GANs) Haoqiang Fan fhq@megvii.com April. 2019 Figures adapted from NIPS 2016](https://reader031.fdocuments.net/reader031/viewer/2022041017/5ecaa2dfe85e912d244177dd/html5/thumbnails/10.jpg)
How Generative Models are Trained
![Page 11: Introduction to Generative Models (and GANs) · 2020-02-06 · Introduction to Generative Models (and GANs) Haoqiang Fan fhq@megvii.com April. 2019 Figures adapted from NIPS 2016](https://reader031.fdocuments.net/reader031/viewer/2022041017/5ecaa2dfe85e912d244177dd/html5/thumbnails/11.jpg)
Learning Generative Models
Figures adapted from NIPS 2016 Tutorial Generative Adversarial Netw orks
![Page 12: Introduction to Generative Models (and GANs) · 2020-02-06 · Introduction to Generative Models (and GANs) Haoqiang Fan fhq@megvii.com April. 2019 Figures adapted from NIPS 2016](https://reader031.fdocuments.net/reader031/viewer/2022041017/5ecaa2dfe85e912d244177dd/html5/thumbnails/12.jpg)
Taxonomy of Generative Models
Figures adapted from NIPS 2016 Tutorial Generative Adversarial Netw orks
![Page 13: Introduction to Generative Models (and GANs) · 2020-02-06 · Introduction to Generative Models (and GANs) Haoqiang Fan fhq@megvii.com April. 2019 Figures adapted from NIPS 2016](https://reader031.fdocuments.net/reader031/viewer/2022041017/5ecaa2dfe85e912d244177dd/html5/thumbnails/13.jpg)
Exact Model: NVP (non-volume preserving)
Density estimation using Real NVP https://arxiv.org/abs/1605.08803
![Page 14: Introduction to Generative Models (and GANs) · 2020-02-06 · Introduction to Generative Models (and GANs) Haoqiang Fan fhq@megvii.com April. 2019 Figures adapted from NIPS 2016](https://reader031.fdocuments.net/reader031/viewer/2022041017/5ecaa2dfe85e912d244177dd/html5/thumbnails/14.jpg)
Real NVP: Invertible Non-linear Transforms
Density estimation using Real NVP
![Page 15: Introduction to Generative Models (and GANs) · 2020-02-06 · Introduction to Generative Models (and GANs) Haoqiang Fan fhq@megvii.com April. 2019 Figures adapted from NIPS 2016](https://reader031.fdocuments.net/reader031/viewer/2022041017/5ecaa2dfe85e912d244177dd/html5/thumbnails/15.jpg)
Real NVP: Examples
Density estimation using Real NVP
![Page 16: Introduction to Generative Models (and GANs) · 2020-02-06 · Introduction to Generative Models (and GANs) Haoqiang Fan fhq@megvii.com April. 2019 Figures adapted from NIPS 2016](https://reader031.fdocuments.net/reader031/viewer/2022041017/5ecaa2dfe85e912d244177dd/html5/thumbnails/16.jpg)
Real NVP
Restriction on the source domain: must be of the same as the target.
![Page 17: Introduction to Generative Models (and GANs) · 2020-02-06 · Introduction to Generative Models (and GANs) Haoqiang Fan fhq@megvii.com April. 2019 Figures adapted from NIPS 2016](https://reader031.fdocuments.net/reader031/viewer/2022041017/5ecaa2dfe85e912d244177dd/html5/thumbnails/17.jpg)
Variational Auto-Encoder
Auto-encoding with noise in
hidden variable
![Page 18: Introduction to Generative Models (and GANs) · 2020-02-06 · Introduction to Generative Models (and GANs) Haoqiang Fan fhq@megvii.com April. 2019 Figures adapted from NIPS 2016](https://reader031.fdocuments.net/reader031/viewer/2022041017/5ecaa2dfe85e912d244177dd/html5/thumbnails/18.jpg)
Variational Auto-Encoder
![Page 19: Introduction to Generative Models (and GANs) · 2020-02-06 · Introduction to Generative Models (and GANs) Haoqiang Fan fhq@megvii.com April. 2019 Figures adapted from NIPS 2016](https://reader031.fdocuments.net/reader031/viewer/2022041017/5ecaa2dfe85e912d244177dd/html5/thumbnails/19.jpg)
![Page 20: Introduction to Generative Models (and GANs) · 2020-02-06 · Introduction to Generative Models (and GANs) Haoqiang Fan fhq@megvii.com April. 2019 Figures adapted from NIPS 2016](https://reader031.fdocuments.net/reader031/viewer/2022041017/5ecaa2dfe85e912d244177dd/html5/thumbnails/20.jpg)
![Page 21: Introduction to Generative Models (and GANs) · 2020-02-06 · Introduction to Generative Models (and GANs) Haoqiang Fan fhq@megvii.com April. 2019 Figures adapted from NIPS 2016](https://reader031.fdocuments.net/reader031/viewer/2022041017/5ecaa2dfe85e912d244177dd/html5/thumbnails/21.jpg)
VAE: Examples
![Page 22: Introduction to Generative Models (and GANs) · 2020-02-06 · Introduction to Generative Models (and GANs) Haoqiang Fan fhq@megvii.com April. 2019 Figures adapted from NIPS 2016](https://reader031.fdocuments.net/reader031/viewer/2022041017/5ecaa2dfe85e912d244177dd/html5/thumbnails/22.jpg)
Generative Adversarial Networks (GAN)
Figures adapted from NIPS 2016 Tutorial Generative Adversarial Netw orks
![Page 23: Introduction to Generative Models (and GANs) · 2020-02-06 · Introduction to Generative Models (and GANs) Haoqiang Fan fhq@megvii.com April. 2019 Figures adapted from NIPS 2016](https://reader031.fdocuments.net/reader031/viewer/2022041017/5ecaa2dfe85e912d244177dd/html5/thumbnails/23.jpg)
DCGAN
Train D by Loss(D(real),1), Loss(D(G(random),0)
Train G by Loss(D(G(random)),1)
http://gluon.mxnet.io/chapter14_generative-adversarial-networks/dcgan.html
![Page 24: Introduction to Generative Models (and GANs) · 2020-02-06 · Introduction to Generative Models (and GANs) Haoqiang Fan fhq@megvii.com April. 2019 Figures adapted from NIPS 2016](https://reader031.fdocuments.net/reader031/viewer/2022041017/5ecaa2dfe85e912d244177dd/html5/thumbnails/24.jpg)
DCGAN: Examples
![Page 25: Introduction to Generative Models (and GANs) · 2020-02-06 · Introduction to Generative Models (and GANs) Haoqiang Fan fhq@megvii.com April. 2019 Figures adapted from NIPS 2016](https://reader031.fdocuments.net/reader031/viewer/2022041017/5ecaa2dfe85e912d244177dd/html5/thumbnails/25.jpg)
DCGAN: Example of Feature Manipulation
Vector arithmetics in feature space
![Page 26: Introduction to Generative Models (and GANs) · 2020-02-06 · Introduction to Generative Models (and GANs) Haoqiang Fan fhq@megvii.com April. 2019 Figures adapted from NIPS 2016](https://reader031.fdocuments.net/reader031/viewer/2022041017/5ecaa2dfe85e912d244177dd/html5/thumbnails/26.jpg)
Conditional, Cross-domain Generation
Generative adversarial text to image synthesis
Figures adapted from NIPS 2016 Tutorial Generative Adversarial Netw orks
![Page 27: Introduction to Generative Models (and GANs) · 2020-02-06 · Introduction to Generative Models (and GANs) Haoqiang Fan fhq@megvii.com April. 2019 Figures adapted from NIPS 2016](https://reader031.fdocuments.net/reader031/viewer/2022041017/5ecaa2dfe85e912d244177dd/html5/thumbnails/27.jpg)
GAN training problems: unstable losses
http://guimperarnau.com/files/blog/Fantastic-GANs-and-where-to-find-them/crazy_loss_function.jpg
![Page 28: Introduction to Generative Models (and GANs) · 2020-02-06 · Introduction to Generative Models (and GANs) Haoqiang Fan fhq@megvii.com April. 2019 Figures adapted from NIPS 2016](https://reader031.fdocuments.net/reader031/viewer/2022041017/5ecaa2dfe85e912d244177dd/html5/thumbnails/28.jpg)
GAN training problems: Mini-batch Fluctuation
Differs much even between consecutive minibatches.
Figures adapted from NIPS 2016 Tutorial Generative Adversarial Netw orks
![Page 29: Introduction to Generative Models (and GANs) · 2020-02-06 · Introduction to Generative Models (and GANs) Haoqiang Fan fhq@megvii.com April. 2019 Figures adapted from NIPS 2016](https://reader031.fdocuments.net/reader031/viewer/2022041017/5ecaa2dfe85e912d244177dd/html5/thumbnails/29.jpg)
GAN training problems: Mode Collapse
Lack of diversity in generated results.
Figures adapted from NIPS 2016 Tutorial Generative Adversarial Netw orks
![Page 30: Introduction to Generative Models (and GANs) · 2020-02-06 · Introduction to Generative Models (and GANs) Haoqiang Fan fhq@megvii.com April. 2019 Figures adapted from NIPS 2016](https://reader031.fdocuments.net/reader031/viewer/2022041017/5ecaa2dfe85e912d244177dd/html5/thumbnails/30.jpg)
Figures adapted from NIPS 2016 Tutorial Generative Adversarial Netw orks
Improve GAN training: Label Smoothing
Improves stability of training
![Page 31: Introduction to Generative Models (and GANs) · 2020-02-06 · Introduction to Generative Models (and GANs) Haoqiang Fan fhq@megvii.com April. 2019 Figures adapted from NIPS 2016](https://reader031.fdocuments.net/reader031/viewer/2022041017/5ecaa2dfe85e912d244177dd/html5/thumbnails/31.jpg)
Improve GAN training: Wasserstein GAN
Use linear instead of log
![Page 32: Introduction to Generative Models (and GANs) · 2020-02-06 · Introduction to Generative Models (and GANs) Haoqiang Fan fhq@megvii.com April. 2019 Figures adapted from NIPS 2016](https://reader031.fdocuments.net/reader031/viewer/2022041017/5ecaa2dfe85e912d244177dd/html5/thumbnails/32.jpg)
WGAN: Stabilized Training Curve
![Page 33: Introduction to Generative Models (and GANs) · 2020-02-06 · Introduction to Generative Models (and GANs) Haoqiang Fan fhq@megvii.com April. 2019 Figures adapted from NIPS 2016](https://reader031.fdocuments.net/reader031/viewer/2022041017/5ecaa2dfe85e912d244177dd/html5/thumbnails/33.jpg)
WGAN: Non-vanishing Gradient
![Page 34: Introduction to Generative Models (and GANs) · 2020-02-06 · Introduction to Generative Models (and GANs) Haoqiang Fan fhq@megvii.com April. 2019 Figures adapted from NIPS 2016](https://reader031.fdocuments.net/reader031/viewer/2022041017/5ecaa2dfe85e912d244177dd/html5/thumbnails/34.jpg)
Loss Sensitive GAN
![Page 35: Introduction to Generative Models (and GANs) · 2020-02-06 · Introduction to Generative Models (and GANs) Haoqiang Fan fhq@megvii.com April. 2019 Figures adapted from NIPS 2016](https://reader031.fdocuments.net/reader031/viewer/2022041017/5ecaa2dfe85e912d244177dd/html5/thumbnails/35.jpg)
![Page 36: Introduction to Generative Models (and GANs) · 2020-02-06 · Introduction to Generative Models (and GANs) Haoqiang Fan fhq@megvii.com April. 2019 Figures adapted from NIPS 2016](https://reader031.fdocuments.net/reader031/viewer/2022041017/5ecaa2dfe85e912d244177dd/html5/thumbnails/36.jpg)
The GAN Zoo
https://github.com/hindupuravinash/the-gan-zoo
![Page 37: Introduction to Generative Models (and GANs) · 2020-02-06 · Introduction to Generative Models (and GANs) Haoqiang Fan fhq@megvii.com April. 2019 Figures adapted from NIPS 2016](https://reader031.fdocuments.net/reader031/viewer/2022041017/5ecaa2dfe85e912d244177dd/html5/thumbnails/37.jpg)
![Page 38: Introduction to Generative Models (and GANs) · 2020-02-06 · Introduction to Generative Models (and GANs) Haoqiang Fan fhq@megvii.com April. 2019 Figures adapted from NIPS 2016](https://reader031.fdocuments.net/reader031/viewer/2022041017/5ecaa2dfe85e912d244177dd/html5/thumbnails/38.jpg)
![Page 39: Introduction to Generative Models (and GANs) · 2020-02-06 · Introduction to Generative Models (and GANs) Haoqiang Fan fhq@megvii.com April. 2019 Figures adapted from NIPS 2016](https://reader031.fdocuments.net/reader031/viewer/2022041017/5ecaa2dfe85e912d244177dd/html5/thumbnails/39.jpg)
![Page 40: Introduction to Generative Models (and GANs) · 2020-02-06 · Introduction to Generative Models (and GANs) Haoqiang Fan fhq@megvii.com April. 2019 Figures adapted from NIPS 2016](https://reader031.fdocuments.net/reader031/viewer/2022041017/5ecaa2dfe85e912d244177dd/html5/thumbnails/40.jpg)
![Page 41: Introduction to Generative Models (and GANs) · 2020-02-06 · Introduction to Generative Models (and GANs) Haoqiang Fan fhq@megvii.com April. 2019 Figures adapted from NIPS 2016](https://reader031.fdocuments.net/reader031/viewer/2022041017/5ecaa2dfe85e912d244177dd/html5/thumbnails/41.jpg)
![Page 42: Introduction to Generative Models (and GANs) · 2020-02-06 · Introduction to Generative Models (and GANs) Haoqiang Fan fhq@megvii.com April. 2019 Figures adapted from NIPS 2016](https://reader031.fdocuments.net/reader031/viewer/2022041017/5ecaa2dfe85e912d244177dd/html5/thumbnails/42.jpg)
![Page 43: Introduction to Generative Models (and GANs) · 2020-02-06 · Introduction to Generative Models (and GANs) Haoqiang Fan fhq@megvii.com April. 2019 Figures adapted from NIPS 2016](https://reader031.fdocuments.net/reader031/viewer/2022041017/5ecaa2dfe85e912d244177dd/html5/thumbnails/43.jpg)
![Page 44: Introduction to Generative Models (and GANs) · 2020-02-06 · Introduction to Generative Models (and GANs) Haoqiang Fan fhq@megvii.com April. 2019 Figures adapted from NIPS 2016](https://reader031.fdocuments.net/reader031/viewer/2022041017/5ecaa2dfe85e912d244177dd/html5/thumbnails/44.jpg)
![Page 45: Introduction to Generative Models (and GANs) · 2020-02-06 · Introduction to Generative Models (and GANs) Haoqiang Fan fhq@megvii.com April. 2019 Figures adapted from NIPS 2016](https://reader031.fdocuments.net/reader031/viewer/2022041017/5ecaa2dfe85e912d244177dd/html5/thumbnails/45.jpg)
Cycle GAN: Correspondence from Unpaired Data
![Page 46: Introduction to Generative Models (and GANs) · 2020-02-06 · Introduction to Generative Models (and GANs) Haoqiang Fan fhq@megvii.com April. 2019 Figures adapted from NIPS 2016](https://reader031.fdocuments.net/reader031/viewer/2022041017/5ecaa2dfe85e912d244177dd/html5/thumbnails/46.jpg)
Cycle GAN
![Page 47: Introduction to Generative Models (and GANs) · 2020-02-06 · Introduction to Generative Models (and GANs) Haoqiang Fan fhq@megvii.com April. 2019 Figures adapted from NIPS 2016](https://reader031.fdocuments.net/reader031/viewer/2022041017/5ecaa2dfe85e912d244177dd/html5/thumbnails/47.jpg)
Cycle GAN: Bad Cases
![Page 48: Introduction to Generative Models (and GANs) · 2020-02-06 · Introduction to Generative Models (and GANs) Haoqiang Fan fhq@megvii.com April. 2019 Figures adapted from NIPS 2016](https://reader031.fdocuments.net/reader031/viewer/2022041017/5ecaa2dfe85e912d244177dd/html5/thumbnails/48.jpg)
DiscoGAN
Cross-domain relation
![Page 49: Introduction to Generative Models (and GANs) · 2020-02-06 · Introduction to Generative Models (and GANs) Haoqiang Fan fhq@megvii.com April. 2019 Figures adapted from NIPS 2016](https://reader031.fdocuments.net/reader031/viewer/2022041017/5ecaa2dfe85e912d244177dd/html5/thumbnails/49.jpg)
DiscoGAN
![Page 50: Introduction to Generative Models (and GANs) · 2020-02-06 · Introduction to Generative Models (and GANs) Haoqiang Fan fhq@megvii.com April. 2019 Figures adapted from NIPS 2016](https://reader031.fdocuments.net/reader031/viewer/2022041017/5ecaa2dfe85e912d244177dd/html5/thumbnails/50.jpg)
How
much
smile?
How
much
smile?
Image A
Image B Reconstructed B
Au
Bε Reconstructed B
Underdetermined CycleGAN pattern
Information Preserving GeneGAN pattern
Smiling from A
Smiling from AAε
Bu
![Page 51: Introduction to Generative Models (and GANs) · 2020-02-06 · Introduction to Generative Models (and GANs) Haoqiang Fan fhq@megvii.com April. 2019 Figures adapted from NIPS 2016](https://reader031.fdocuments.net/reader031/viewer/2022041017/5ecaa2dfe85e912d244177dd/html5/thumbnails/51.jpg)
GeneGAN: shorter pathway improves training
Cross breeds and reproductions
![Page 52: Introduction to Generative Models (and GANs) · 2020-02-06 · Introduction to Generative Models (and GANs) Haoqiang Fan fhq@megvii.com April. 2019 Figures adapted from NIPS 2016](https://reader031.fdocuments.net/reader031/viewer/2022041017/5ecaa2dfe85e912d244177dd/html5/thumbnails/52.jpg)
GeneGAN: Object Transfiguration
Transfer "my" hairstyle to him, not just a hairstyle.
Slide
Github
![Page 53: Introduction to Generative Models (and GANs) · 2020-02-06 · Introduction to Generative Models (and GANs) Haoqiang Fan fhq@megvii.com April. 2019 Figures adapted from NIPS 2016](https://reader031.fdocuments.net/reader031/viewer/2022041017/5ecaa2dfe85e912d244177dd/html5/thumbnails/53.jpg)
Math behind Generative Models
Those who don’t care about math or theory can open their
PyTorch now...
![Page 54: Introduction to Generative Models (and GANs) · 2020-02-06 · Introduction to Generative Models (and GANs) Haoqiang Fan fhq@megvii.com April. 2019 Figures adapted from NIPS 2016](https://reader031.fdocuments.net/reader031/viewer/2022041017/5ecaa2dfe85e912d244177dd/html5/thumbnails/54.jpg)
Formulation of Generative Models
sampling v.s. density estimation
![Page 55: Introduction to Generative Models (and GANs) · 2020-02-06 · Introduction to Generative Models (and GANs) Haoqiang Fan fhq@megvii.com April. 2019 Figures adapted from NIPS 2016](https://reader031.fdocuments.net/reader031/viewer/2022041017/5ecaa2dfe85e912d244177dd/html5/thumbnails/55.jpg)
RBM
![Page 56: Introduction to Generative Models (and GANs) · 2020-02-06 · Introduction to Generative Models (and GANs) Haoqiang Fan fhq@megvii.com April. 2019 Figures adapted from NIPS 2016](https://reader031.fdocuments.net/reader031/viewer/2022041017/5ecaa2dfe85e912d244177dd/html5/thumbnails/56.jpg)
RBM
It is NP-Hard to
estimate Z
![Page 57: Introduction to Generative Models (and GANs) · 2020-02-06 · Introduction to Generative Models (and GANs) Haoqiang Fan fhq@megvii.com April. 2019 Figures adapted from NIPS 2016](https://reader031.fdocuments.net/reader031/viewer/2022041017/5ecaa2dfe85e912d244177dd/html5/thumbnails/57.jpg)
RBM
It is NP-Hard to
sample from P
![Page 58: Introduction to Generative Models (and GANs) · 2020-02-06 · Introduction to Generative Models (and GANs) Haoqiang Fan fhq@megvii.com April. 2019 Figures adapted from NIPS 2016](https://reader031.fdocuments.net/reader031/viewer/2022041017/5ecaa2dfe85e912d244177dd/html5/thumbnails/58.jpg)
Score Matching
Let L be the likelihood function, score V is:
If two distribution’s scores match, they also match.
![Page 59: Introduction to Generative Models (and GANs) · 2020-02-06 · Introduction to Generative Models (and GANs) Haoqiang Fan fhq@megvii.com April. 2019 Figures adapted from NIPS 2016](https://reader031.fdocuments.net/reader031/viewer/2022041017/5ecaa2dfe85e912d244177dd/html5/thumbnails/59.jpg)
Markov Chain Monte Carlo
From each node a,
walk to “neighbor” b with probability proportional to p(b).
Neighbors must be reciprocal: a <->b
Walk for long enough time to reach equilibrium
a b
p(a)/p(b)/N
1/N
![Page 60: Introduction to Generative Models (and GANs) · 2020-02-06 · Introduction to Generative Models (and GANs) Haoqiang Fan fhq@megvii.com April. 2019 Figures adapted from NIPS 2016](https://reader031.fdocuments.net/reader031/viewer/2022041017/5ecaa2dfe85e912d244177dd/html5/thumbnails/60.jpg)
MCMC in RBM
Sample x given y
Sample y given x
Sample x given y
…..
In theory, repeat for long enough time.
In practice, repeat a few times. ("burnin")
![Page 61: Introduction to Generative Models (and GANs) · 2020-02-06 · Introduction to Generative Models (and GANs) Haoqiang Fan fhq@megvii.com April. 2019 Figures adapted from NIPS 2016](https://reader031.fdocuments.net/reader031/viewer/2022041017/5ecaa2dfe85e912d244177dd/html5/thumbnails/61.jpg)
RBM: Learned “Filters”
![Page 62: Introduction to Generative Models (and GANs) · 2020-02-06 · Introduction to Generative Models (and GANs) Haoqiang Fan fhq@megvii.com April. 2019 Figures adapted from NIPS 2016](https://reader031.fdocuments.net/reader031/viewer/2022041017/5ecaa2dfe85e912d244177dd/html5/thumbnails/62.jpg)
From Density to Sample
Given density function p(x), can we efficiently black-box sample from it?
No! p(x)= MD5(x)==0
Unless query Ω(N) samples, it is hard to determine.
![Page 63: Introduction to Generative Models (and GANs) · 2020-02-06 · Introduction to Generative Models (and GANs) Haoqiang Fan fhq@megvii.com April. 2019 Figures adapted from NIPS 2016](https://reader031.fdocuments.net/reader031/viewer/2022041017/5ecaa2dfe85e912d244177dd/html5/thumbnails/63.jpg)
From Sample to Density
Given black-box sampler G, can we efficiently estimate the density (frequency) of
x?
Naive bound: Ω(ε-2) absolute, Ω(1/p(x) ε-2) relative
Cannot essentially do better.
Example: Sample x randomly. Retry iff x=0.
![Page 64: Introduction to Generative Models (and GANs) · 2020-02-06 · Introduction to Generative Models (and GANs) Haoqiang Fan fhq@megvii.com April. 2019 Figures adapted from NIPS 2016](https://reader031.fdocuments.net/reader031/viewer/2022041017/5ecaa2dfe85e912d244177dd/html5/thumbnails/64.jpg)
What can be done if only samples are available?
Problem: Given black box sampler G, decide if:
(1) it is uniform
(2) it is ε-far from uniform
How to define distance between distributions?
Statistical distance: ½ sum |p(x)-q(x)| p:G q:Uniform
L2 distance: sum (p(x)-q(x))2
KL divergence: sum q(x)log(q(x)/p(x))
![Page 65: Introduction to Generative Models (and GANs) · 2020-02-06 · Introduction to Generative Models (and GANs) Haoqiang Fan fhq@megvii.com April. 2019 Figures adapted from NIPS 2016](https://reader031.fdocuments.net/reader031/viewer/2022041017/5ecaa2dfe85e912d244177dd/html5/thumbnails/65.jpg)
Uniformity Check using q(x)log(q(x)/p(x))
Impossible to check unless Ω(N) samples are obtained.
Consider {1,2,...,N}T and {1,2,...,N-1}T. Unbound KL.
Statistical distance = sum max(p(x)-q(x),0)
((N-1)/N)T = 1-o(1) if T=o(N)
Statistical distance is the best distinguisher’s advantage over random
guess!
advantage = 2*|Pr(guess correct)-0.5|
![Page 66: Introduction to Generative Models (and GANs) · 2020-02-06 · Introduction to Generative Models (and GANs) Haoqiang Fan fhq@megvii.com April. 2019 Figures adapted from NIPS 2016](https://reader031.fdocuments.net/reader031/viewer/2022041017/5ecaa2dfe85e912d244177dd/html5/thumbnails/66.jpg)
Uniformity Check using L2 Distance
sum (p(x)-q(x))2 = sum p(x)2+q(x)2-2p(x)q(x) = sum p(x)2 - 1/N
p(x)2 : seeing two x in a row
sum p(x)2: counting collisions
Algorithm: Get T samples, count the number of x[i]==x[j] for i<j, divide by C(T,2)
variance calculation: O(ε2) is enough!
![Page 67: Introduction to Generative Models (and GANs) · 2020-02-06 · Introduction to Generative Models (and GANs) Haoqiang Fan fhq@megvii.com April. 2019 Figures adapted from NIPS 2016](https://reader031.fdocuments.net/reader031/viewer/2022041017/5ecaa2dfe85e912d244177dd/html5/thumbnails/67.jpg)
Uniformity Check using L1 Distance
Estimate collision probability to 1±O(ε2)
O(ε-4sqrt(N)) samples are enough.
![Page 68: Introduction to Generative Models (and GANs) · 2020-02-06 · Introduction to Generative Models (and GANs) Haoqiang Fan fhq@megvii.com April. 2019 Figures adapted from NIPS 2016](https://reader031.fdocuments.net/reader031/viewer/2022041017/5ecaa2dfe85e912d244177dd/html5/thumbnails/68.jpg)
Lessons Learned: What We Can Get From Samples
Given samples, some properties of the distribution can be learned, while others
cannot.
![Page 69: Introduction to Generative Models (and GANs) · 2020-02-06 · Introduction to Generative Models (and GANs) Haoqiang Fan fhq@megvii.com April. 2019 Figures adapted from NIPS 2016](https://reader031.fdocuments.net/reader031/viewer/2022041017/5ecaa2dfe85e912d244177dd/html5/thumbnails/69.jpg)
Discriminator based distances
maxD E(D(x))x~p - E(D(y))y~q
0<=D<=1 : Statistical Distance
D is Lipschitz Continuous: Wasserstein Distance
![Page 70: Introduction to Generative Models (and GANs) · 2020-02-06 · Introduction to Generative Models (and GANs) Haoqiang Fan fhq@megvii.com April. 2019 Figures adapted from NIPS 2016](https://reader031.fdocuments.net/reader031/viewer/2022041017/5ecaa2dfe85e912d244177dd/html5/thumbnails/70.jpg)
Wasserstein Distance
Duality
Earth Mover Distance:
Definition using Discriminator:
![Page 71: Introduction to Generative Models (and GANs) · 2020-02-06 · Introduction to Generative Models (and GANs) Haoqiang Fan fhq@megvii.com April. 2019 Figures adapted from NIPS 2016](https://reader031.fdocuments.net/reader031/viewer/2022041017/5ecaa2dfe85e912d244177dd/html5/thumbnails/71.jpg)
Estimating Wasserstein Distance in High Dimension
The curse of dimensionality
There is no algorithm that, for any two distributions P and Q in an n-dimensional
space with radius r,
takes poly(n) samples from P and Q and estimates W(P,Q) to precision o(1)*r
w.h.p.
![Page 72: Introduction to Generative Models (and GANs) · 2020-02-06 · Introduction to Generative Models (and GANs) Haoqiang Fan fhq@megvii.com April. 2019 Figures adapted from NIPS 2016](https://reader031.fdocuments.net/reader031/viewer/2022041017/5ecaa2dfe85e912d244177dd/html5/thumbnails/72.jpg)
Finite Sample Version of EMD
Let WN(P,Q) be the expected EMD between N samples from P and Q.
WN(P,Q)>=W(P,Q)
W(P,Q)≥WN(P,Q)-min(WN(P,P),WN(Q,Q))
![Page 73: Introduction to Generative Models (and GANs) · 2020-02-06 · Introduction to Generative Models (and GANs) Haoqiang Fan fhq@megvii.com April. 2019 Figures adapted from NIPS 2016](https://reader031.fdocuments.net/reader031/viewer/2022041017/5ecaa2dfe85e912d244177dd/html5/thumbnails/73.jpg)
Projected Wasserstein Distance
The k-dimensional projected EMD: let σ be a random k-dim subspace
As a lower bounding approach
![Page 74: Introduction to Generative Models (and GANs) · 2020-02-06 · Introduction to Generative Models (and GANs) Haoqiang Fan fhq@megvii.com April. 2019 Figures adapted from NIPS 2016](https://reader031.fdocuments.net/reader031/viewer/2022041017/5ecaa2dfe85e912d244177dd/html5/thumbnails/74.jpg)
Game Theory: The Generator - Discriminator Game
Stackelberg Game:
min. D max. G
min. G max. D
Nash equilibrium
(G,D) where both G and D will not deviate
Which is the largest?
![Page 75: Introduction to Generative Models (and GANs) · 2020-02-06 · Introduction to Generative Models (and GANs) Haoqiang Fan fhq@megvii.com April. 2019 Figures adapted from NIPS 2016](https://reader031.fdocuments.net/reader031/viewer/2022041017/5ecaa2dfe85e912d244177dd/html5/thumbnails/75.jpg)
Linear Model
minimax theorem
![Page 76: Introduction to Generative Models (and GANs) · 2020-02-06 · Introduction to Generative Models (and GANs) Haoqiang Fan fhq@megvii.com April. 2019 Figures adapted from NIPS 2016](https://reader031.fdocuments.net/reader031/viewer/2022041017/5ecaa2dfe85e912d244177dd/html5/thumbnails/76.jpg)
The Future of GANs
Guaranteed stabilization: new distance
Broader application: apply adversarial loss in XX / different type of data
![Page 77: Introduction to Generative Models (and GANs) · 2020-02-06 · Introduction to Generative Models (and GANs) Haoqiang Fan fhq@megvii.com April. 2019 Figures adapted from NIPS 2016](https://reader031.fdocuments.net/reader031/viewer/2022041017/5ecaa2dfe85e912d244177dd/html5/thumbnails/77.jpg)
References
GAN Tutorial: https://arxiv.org/pdf/1701.00160.pdf
Slides: https://media.nips.cc/Conferences/2016/Slides/6202-Slides.pdf
![Page 78: Introduction to Generative Models (and GANs) · 2020-02-06 · Introduction to Generative Models (and GANs) Haoqiang Fan fhq@megvii.com April. 2019 Figures adapted from NIPS 2016](https://reader031.fdocuments.net/reader031/viewer/2022041017/5ecaa2dfe85e912d244177dd/html5/thumbnails/78.jpg)
Backup after this slide
![Page 79: Introduction to Generative Models (and GANs) · 2020-02-06 · Introduction to Generative Models (and GANs) Haoqiang Fan fhq@megvii.com April. 2019 Figures adapted from NIPS 2016](https://reader031.fdocuments.net/reader031/viewer/2022041017/5ecaa2dfe85e912d244177dd/html5/thumbnails/79.jpg)
GeneGAN: Interpolation in Object Subspace
Check the directions of the hairs.
Bi-linearly interpolated
ε instance