High-Fidelity Image Generation With *equal contribution ...11-14-00)-11-14-25-4622... ·...
Transcript of High-Fidelity Image Generation With *equal contribution ...11-14-00)-11-14-25-4622... ·...
![Page 1: High-Fidelity Image Generation With *equal contribution ...11-14-00)-11-14-25-4622... · High-Fidelity Image Generation With Fewer Labels Michael Tschannen* Mario Lucic* Marvin Ritter*](https://reader033.fdocuments.net/reader033/viewer/2022041416/5e1bcaffaf679111be4a8053/html5/thumbnails/1.jpg)
High-Fidelity Image Generation With Fewer LabelsMichael Tschannen*
Mario Lucic* Marvin Ritter* Xiaohua Zhai Olivier Bachem Sylvain Gelly
*equal contribution
![Page 2: High-Fidelity Image Generation With *equal contribution ...11-14-00)-11-14-25-4622... · High-Fidelity Image Generation With Fewer Labels Michael Tschannen* Mario Lucic* Marvin Ritter*](https://reader033.fdocuments.net/reader033/viewer/2022041416/5e1bcaffaf679111be4a8053/html5/thumbnails/2.jpg)
Generative Adversarial Networks (GANs): Recent Progress
P 3
BigGAN (Brock, Donahue, Simonyan 2019)
![Page 3: High-Fidelity Image Generation With *equal contribution ...11-14-00)-11-14-25-4622... · High-Fidelity Image Generation With Fewer Labels Michael Tschannen* Mario Lucic* Marvin Ritter*](https://reader033.fdocuments.net/reader033/viewer/2022041416/5e1bcaffaf679111be4a8053/html5/thumbnails/3.jpg)
Generative Adversarial Networks (GANs): Recent Progress
P 4
BigGAN (Brock, Donahue, Simonyan 2019)class-conditional
![Page 4: High-Fidelity Image Generation With *equal contribution ...11-14-00)-11-14-25-4622... · High-Fidelity Image Generation With Fewer Labels Michael Tschannen* Mario Lucic* Marvin Ritter*](https://reader033.fdocuments.net/reader033/viewer/2022041416/5e1bcaffaf679111be4a8053/html5/thumbnails/4.jpg)
Generative Adversarial Networks (GANs): Recent Progress
Conditioning reduces the diverse generation problem to a per-class problem
P 5
BigGAN (Brock, Donahue, Simonyan 2019)class-conditional
![Page 5: High-Fidelity Image Generation With *equal contribution ...11-14-00)-11-14-25-4622... · High-Fidelity Image Generation With Fewer Labels Michael Tschannen* Mario Lucic* Marvin Ritter*](https://reader033.fdocuments.net/reader033/viewer/2022041416/5e1bcaffaf679111be4a8053/html5/thumbnails/5.jpg)
Generative Adversarial Networks (GANs): Recent Progress
Conditioning reduces the diverse generation problem to a per-class problem
P 6
BigGAN (Brock, Donahue, Simonyan 2019) SS-GAN (Chen et al. 2019)class-conditional unsupervised
![Page 6: High-Fidelity Image Generation With *equal contribution ...11-14-00)-11-14-25-4622... · High-Fidelity Image Generation With Fewer Labels Michael Tschannen* Mario Lucic* Marvin Ritter*](https://reader033.fdocuments.net/reader033/viewer/2022041416/5e1bcaffaf679111be4a8053/html5/thumbnails/6.jpg)
Generative Adversarial Networks (GANs): Recent Progress
Conditioning reduces the diverse generation problem to a per-class problem
P 7
BigGAN (Brock, Donahue, Simonyan 2019) SS-GAN (Chen et al. 2019)class-conditional unsupervised
Unsupervised models are considerably less powerful
![Page 7: High-Fidelity Image Generation With *equal contribution ...11-14-00)-11-14-25-4622... · High-Fidelity Image Generation With Fewer Labels Michael Tschannen* Mario Lucic* Marvin Ritter*](https://reader033.fdocuments.net/reader033/viewer/2022041416/5e1bcaffaf679111be4a8053/html5/thumbnails/7.jpg)
This work: How to close the gap between conditional and unsupervised GANs?
Generative Adversarial Networks (GANs): Recent Progress
P 8
BigGAN (Brock, Donahue, Simonyan 2019) SS-GAN (Chen et al. 2019)class-conditional unsupervised
![Page 8: High-Fidelity Image Generation With *equal contribution ...11-14-00)-11-14-25-4622... · High-Fidelity Image Generation With Fewer Labels Michael Tschannen* Mario Lucic* Marvin Ritter*](https://reader033.fdocuments.net/reader033/viewer/2022041416/5e1bcaffaf679111be4a8053/html5/thumbnails/8.jpg)
Proposed methods: Overview
P 9
● Replace ground-truth labels with synthetic/inferred labels➜ No changes in the GAN architecture required
● Infer labels for the real data using self-supervised and semi-supervised learning techniques
![Page 9: High-Fidelity Image Generation With *equal contribution ...11-14-00)-11-14-25-4622... · High-Fidelity Image Generation With Fewer Labels Michael Tschannen* Mario Lucic* Marvin Ritter*](https://reader033.fdocuments.net/reader033/viewer/2022041416/5e1bcaffaf679111be4a8053/html5/thumbnails/9.jpg)
Proposed methods: Pre-training
P 10
1. Learn a semantic representation F of the data using self-supervision by rotation prediction (Gidaris et al. 2018)
2. Clustering or semi-supervised learning on the representation F 3. Train GAN with inferred labels
![Page 10: High-Fidelity Image Generation With *equal contribution ...11-14-00)-11-14-25-4622... · High-Fidelity Image Generation With Fewer Labels Michael Tschannen* Mario Lucic* Marvin Ritter*](https://reader033.fdocuments.net/reader033/viewer/2022041416/5e1bcaffaf679111be4a8053/html5/thumbnails/10.jpg)
Proposed methods: Co-training
P 11
● Semi-supervised classification head on discriminator
![Page 11: High-Fidelity Image Generation With *equal contribution ...11-14-00)-11-14-25-4622... · High-Fidelity Image Generation With Fewer Labels Michael Tschannen* Mario Lucic* Marvin Ritter*](https://reader033.fdocuments.net/reader033/viewer/2022041416/5e1bcaffaf679111be4a8053/html5/thumbnails/11.jpg)
Improve pre- and co-training methods
P 12
● Rotation-self supervision during GAN training (Chen et al. 2019)
![Page 12: High-Fidelity Image Generation With *equal contribution ...11-14-00)-11-14-25-4622... · High-Fidelity Image Generation With Fewer Labels Michael Tschannen* Mario Lucic* Marvin Ritter*](https://reader033.fdocuments.net/reader033/viewer/2022041416/5e1bcaffaf679111be4a8053/html5/thumbnails/12.jpg)
● Clustering (SS) is unsupervised SOTA (FID 22.0)● S2GAN (20%) and S3GAN (10%) match BigGAN (100%)● S3GAN (20%) outperforms BigGAN (100%) (SOTA)
Results
P 13
BigGAN (100%)
![Page 13: High-Fidelity Image Generation With *equal contribution ...11-14-00)-11-14-25-4622... · High-Fidelity Image Generation With Fewer Labels Michael Tschannen* Mario Lucic* Marvin Ritter*](https://reader033.fdocuments.net/reader033/viewer/2022041416/5e1bcaffaf679111be4a8053/html5/thumbnails/13.jpg)
Samples: BigGAN (our implementation) vs proposed
P 14
S3GAN (10%)
BigGAN (100%)
256 x 256 px
![Page 14: High-Fidelity Image Generation With *equal contribution ...11-14-00)-11-14-25-4622... · High-Fidelity Image Generation With Fewer Labels Michael Tschannen* Mario Lucic* Marvin Ritter*](https://reader033.fdocuments.net/reader033/viewer/2022041416/5e1bcaffaf679111be4a8053/html5/thumbnails/14.jpg)
Results
P 16S3GAN (10%) 256 x 256 px
![Page 15: High-Fidelity Image Generation With *equal contribution ...11-14-00)-11-14-25-4622... · High-Fidelity Image Generation With Fewer Labels Michael Tschannen* Mario Lucic* Marvin Ritter*](https://reader033.fdocuments.net/reader033/viewer/2022041416/5e1bcaffaf679111be4a8053/html5/thumbnails/15.jpg)
Code, pretrained models and Colabs:
github.com/google/compare_gan
Check out our poster #13 tonight 6:30-9:00 pm!
P 17