Deep Advances in Generative Modeling

27
Deep Advances in Generative Modeling Alec Radford @AlecRad March 5 th 2016

Transcript of Deep Advances in Generative Modeling

Page 1: Deep Advances in Generative Modeling

Deep Advances in Generative Modeling

Alec Radford@AlecRad

March 5th 2016

https://twitter.com/alecrad
Page 2: Deep Advances in Generative Modeling

Generative modelingModeling complex high dimensional data is an open problem.

Deep generative models are currently making progress here.

Various areas of study/application:

unsupervised/representation/manifold learning

generative counterparts of discriminative models

density/likelihood estimation

conditional generation

Page 3: Deep Advances in Generative Modeling

Examples of Generative Modeling

Page 4: Deep Advances in Generative Modeling

CNNs and RNNs

Page 5: Deep Advances in Generative Modeling

Useful Generative Model - Skipthought [1506.06726]

Page 6: Deep Advances in Generative Modeling

Two promising approaches

Variational Autoencoders (VAE) Kingma and Welling [1312.6114]

Generative Adversarial Networks (GAN) Goodfellow et al. [1406.2661]

encoder Z decoder x̂X

z generator x̂discriminato

r

X

http://arxiv.org/abs/1312.6114
http://arxiv.org/abs/1312.6114
http://arxiv.org/abs/1406.2661
Page 7: Deep Advances in Generative Modeling

Variational Autoencoder

from Kingma and Welling [1312.6114]

● Theoretically elegant autoencoder● Straightforward to implement● Impose a prior on code space

○ regularization○ allows for sampling

● Optimizes variational lower bound on likelihood

encoder Z decoder x̂X

http://arxiv.org/abs/1312.6114
Page 8: Deep Advances in Generative Modeling

Generative Adversarial Networks z x̂

discriminator

X

generator

Page 9: Deep Advances in Generative Modeling

Generative Adversarial Networks z x̂

discriminator

X

generator

Page 10: Deep Advances in Generative Modeling

VAE Extensions

from Kingma et al. [1406.5298]

Semi-Supervised Learning

from Gregor et al. [1502.04623]

DRAW

http://arxiv.org/abs/1406.5298
http://arxiv.org/abs/1502.04623
Page 11: Deep Advances in Generative Modeling

GAN Extensions - LAPGAN

Page 12: Deep Advances in Generative Modeling

Deep convolutional GANs (DCGAN) [1511.06434]

Luke Metz Soumith ChintalaAlec Radford

tl;dr add more layers

indico indico FAIR

Page 13: Deep Advances in Generative Modeling

Deep convolutional GANs (DCGAN) [1511.06434]

Page 14: Deep Advances in Generative Modeling

DCGAN Architecture tricks

No fully connected layers

Batch Normalization Ioffe and Szegedy [1502.03167]

Leaky Rectifier in the discriminator

Use Adam Kingma and Ba [1412.6980]

Tweak Adam hyperparameters a bit (lr=0.0002, b1=0.5)

http://arxiv.org/abs/1502.03167
http://arxiv.org/abs/1412.6980
Page 15: Deep Advances in Generative Modeling

Really really really ridiculously good looking samples

on constrained image distributions :(

Page 16: Deep Advances in Generative Modeling

Interpolation suggests non-overfitting behavior

Page 17: Deep Advances in Generative Modeling

Vector arithmetic properties of generator

Page 18: Deep Advances in Generative Modeling

Generator disentangles objects from scene?

Page 19: Deep Advances in Generative Modeling

Discriminator learns generalizing object detectors

These are responses on validation examples!

Page 20: Deep Advances in Generative Modeling

Results on standard supervised tasks

Page 21: Deep Advances in Generative Modeling

Conditional DCGAN

Page 22: Deep Advances in Generative Modeling

Conditional DCGAN (unpublished)

Sunrise over the ocean

Beautiful falls and stream

sahara desert sand dunes

Tropical rainforest brazil

Stars of the milkyway at night

Page 23: Deep Advances in Generative Modeling

IssuesStill not completely stable

especially for deep and higher res

Unconstrained natural images

Even the biggest models underfit

Hard to evaluate

no reliable/straightforward metrics

No inference model

limits kinds of analysis

Little work on conv VAE equivalents

makes comparison difficult

Some funky stuff going on

separate data/sample batchnorm statistics

train with heuristic cost not GAN theory

Page 24: Deep Advances in Generative Modeling

Hybridizing VAEs and GANs (best of both worlds?)

from Larsen et. al [1512.09300] from Larsen et. al [1512.09300]

http://arxiv.org/pdf/1512.09300
http://arxiv.org/pdf/1512.09300
Page 25: Deep Advances in Generative Modeling

Hybridizing VAEs and GANs (best of both worlds?)

from Larsen et. al [1512.09300] from Larsen et. al [1512.09300]

http://arxiv.org/pdf/1512.09300
http://arxiv.org/pdf/1512.09300
Page 26: Deep Advances in Generative Modeling

Thanks!

Questions?

Page 27: Deep Advances in Generative Modeling

indico.io


Deep Generative models for Inverse Problemscorelab.ntua.gr/ml-seminar/slides/Dimakis_NTUA.pdf · Deep Generative models for Inverse Problems. Outline • Generative Models • Using

Deep Generative models for Inverse Problemscorelab.ntua.gr/ml-seminar/slides/Dimakis_NTUA.pdf · Deep Generative models for Inverse Problems. Outline • Generative Models • Using

Learning Deep Generative Models - MIT

Learning Deep Generative Models - MIT

Supplemental Material for: Deep Generative Stochastic ...s.yosinski.com/Bengio__2014__Deep_Generative... · Supplemental Material for: Deep Generative Stochastic Networks Trainable

Supplemental Material for: Deep Generative Stochastic ...s.yosinski.com/Bengio__2014__Deep_Generative... · Supplemental Material for: Deep Generative Stochastic Networks Trainable

Deep Generative Models with Learnable Knowledge Constraintspapers.nips.cc/paper/8250-deep-generative-models-with-learnable... · Deep Generative Models with Learnable Knowledge Constraints

Deep Generative Models with Learnable Knowledge Constraintspapers.nips.cc/paper/8250-deep-generative-models-with-learnable... · Deep Generative Models with Learnable Knowledge Constraints

Tutorial on Deep Generative Models - shakirm.com · Abstract This tutorial will be a review of recent advances in deep generative models. Generative models have a long history at

Tutorial on Deep Generative Models - shakirm.com · Abstract This tutorial will be a review of recent advances in deep generative models. Generative models have a long history at

Generative chemistry: drug discovery with deep learning … · 2020. 8. 21. · 1 Generative chemistry: drug discovery with deep learning generative models Yuemin Bian1,2 and Xiang-Qun

Generative chemistry: drug discovery with deep learning … · 2020. 8. 21. · 1 Generative chemistry: drug discovery with deep learning generative models Yuemin Bian1,2 and Xiang-Qun

Automatic Colorization with Deep Convolutional Generative ...

Automatic Colorization with Deep Convolutional Generative ...

Unsupervised Image Reconstruction using Deep Generative ...web.stanford.edu/~ekcole/sedona.pdfUnsupervised Image Reconstruction using Deep Generative Adversarial Networks Elizabeth

Unsupervised Image Reconstruction using Deep Generative ...web.stanford.edu/~ekcole/sedona.pdfUnsupervised Image Reconstruction using Deep Generative Adversarial Networks Elizabeth

Deep Generative Modelsslwang/generative_model.pdf · minimization." Advances in Neural Information Processing Systems. 2016. [6] Denton, Emily L., Soumith Chintala, and Rob Fergus.

Deep Generative Modelsslwang/generative_model.pdf · minimization." Advances in Neural Information Processing Systems. 2016. [6] Denton, Emily L., Soumith Chintala, and Rob Fergus.

Deep Generative Models - WordPress.com to a wide variety of problems giving state-of-the-art results in image ... Advances in deep generative models are at the forefront of deep learning

Deep Generative Models - WordPress.com to a wide variety of problems giving state-of-the-art results in image ... Advances in deep generative models are at the forefront of deep learning

Deep Generative Design: Integration of Topology ... · Deep Generative Design: Integration of Topology Optimization and Generative Models ... design exploration and shape parameterization

Deep Generative Design: Integration of Topology ... · Deep Generative Design: Integration of Topology Optimization and Generative Models ... design exploration and shape parameterization

Deep Feedforward Generative Models - GitHub Pagesaliensunmin.github.io/project/accv16tutorial/media/generative.pdf · •We will focus on deep feedforward generative models. ... perceptual

Deep Feedforward Generative Models - GitHub Pagesaliensunmin.github.io/project/accv16tutorial/media/generative.pdf · •We will focus on deep feedforward generative models. ... perceptual

A Deep Generative Framework for Paraphrase Generation · 2019-08-15 · eration problem. However, recent advances in deep learn-ing, in particular generative models (Bowman et al.

A Deep Generative Framework for Paraphrase Generation · 2019-08-15 · eration problem. However, recent advances in deep learn-ing, in particular generative models (Bowman et al.

Interpretable Deep Generative Recommendation Models

Interpretable Deep Generative Recommendation Models

Deep Generative Design: Integration of Topology ...

Deep Generative Design: Integration of Topology ...

Deep Generative Models I

Deep Generative Models I

Deep generative networks as models ofalgonauts.csail.mit.edu/slides/Algonauts2019_Thomas_Naselaris.pdf · Algonauts Workshop —July 19, 2019 —MIT Deep generative networks as models

Deep generative networks as models ofalgonauts.csail.mit.edu/slides/Algonauts2019_Thomas_Naselaris.pdf · Algonauts Workshop —July 19, 2019 —MIT Deep generative networks as models

Deep generative learning_icml_part1

Deep generative learning_icml_part1

Deep Generative Models with Stick-Breaking Priorsenalisni/nalisnick_uciAIML_talk.pdf · Overview of Deep Generative Models. Deep Generative Models with Stick-Breaking Priors. 1 3

Deep Generative Models with Stick-Breaking Priorsenalisni/nalisnick_uciAIML_talk.pdf · Overview of Deep Generative Models. Deep Generative Models with Stick-Breaking Priors. 1 3

Deep Generative Markov State Modelspapers.nips.cc/paper/7653-deep-generative-markov-state... · 2019-02-19 · Deep Generative Markov State Models Hao Wu 1,2 ∗, Andreas Mardt ,

Deep Generative Markov State Modelspapers.nips.cc/paper/7653-deep-generative-markov-state... · 2019-02-19 · Deep Generative Markov State Models Hao Wu 1,2 ∗, Andreas Mardt ,