Михаил Ульянов. Возвращаясь к самому себе.5 Надо полагать, что пик мой творческий — позади. Началось
Дмитрий Ульянов, SKOLTECH - MIXAR2016
-
Upload
mixarconference -
Category
Education
-
view
111 -
download
9
Transcript of Дмитрий Ульянов, SKOLTECH - MIXAR2016
![Page 1: Дмитрий Ульянов, SKOLTECH - MIXAR2016](https://reader034.fdocuments.net/reader034/viewer/2022051007/587fc4ea1a28ab3b158b5cd7/html5/thumbnails/1.jpg)
Image Style Transfer,Neural Doodles &Texture Synthesis
Dmitry Ulyanov
11/13/2016 1
MIXAR
Moscow, 2016
![Page 2: Дмитрий Ульянов, SKOLTECH - MIXAR2016](https://reader034.fdocuments.net/reader034/viewer/2022051007/587fc4ea1a28ab3b158b5cd7/html5/thumbnails/2.jpg)
Dmitry Ulyanov
• Consist of repeated• Convolutions
• ReLU
• MaxPool
+
• FC + Softmax at the end
• Activations (feature maps)
• Tensor of size CxWxH
VGG-style neural networks
11/13/2016 Style transfer, neural doodles & texture synthesis 2
Image credit: Xavier Giro, DeepFix slides
H
W
C
![Page 3: Дмитрий Ульянов, SKOLTECH - MIXAR2016](https://reader034.fdocuments.net/reader034/viewer/2022051007/587fc4ea1a28ab3b158b5cd7/html5/thumbnails/3.jpg)
Dmitry Ulyanov
Image generation examples
11/13/2016 Style transfer, neural doodles & texture synthesis 3
Simonyan et al. 2014Mordvintsev, 2015
![Page 4: Дмитрий Ульянов, SKOLTECH - MIXAR2016](https://reader034.fdocuments.net/reader034/viewer/2022051007/587fc4ea1a28ab3b158b5cd7/html5/thumbnails/4.jpg)
Dmitry Ulyanov
• General overview:1. Texture synthesis
2. Image style transfer
3. Neural doodles
• Our work “Texture networks” (ICML 2016):• Fast texture synthesis
• Fast image style transfer
• Fast neural doodles
Presentation structure
11/13/2016 Style transfer, neural doodles & texture synthesis 4
![Page 5: Дмитрий Ульянов, SKOLTECH - MIXAR2016](https://reader034.fdocuments.net/reader034/viewer/2022051007/587fc4ea1a28ab3b158b5cd7/html5/thumbnails/5.jpg)
Dmitry Ulyanov
Examples: Texture Synthesis
11/13/2016 Style transfer, neural doodles & texture synthesis 5
Source Synthesized
L. A. Gatys, A. S. Ecker, M. Bethge; “Texture Synthesis Using Convolutional Neural Networks“; NIPS 2015
![Page 6: Дмитрий Ульянов, SKOLTECH - MIXAR2016](https://reader034.fdocuments.net/reader034/viewer/2022051007/587fc4ea1a28ab3b158b5cd7/html5/thumbnails/6.jpg)
Dmitry Ulyanov
Examples: Image Artistic Style Transfer
11/13/2016 Style transfer, neural doodles & texture synthesis 6
L. A. Gatys, A. S. Ecker, M. Bethge; “Image Style Transfer Using Convolutional Neural Networks“; CVPR 2016
Content Style Result
![Page 7: Дмитрий Ульянов, SKOLTECH - MIXAR2016](https://reader034.fdocuments.net/reader034/viewer/2022051007/587fc4ea1a28ab3b158b5cd7/html5/thumbnails/7.jpg)
Dmitry Ulyanov
Examples: Neural Doodles
11/13/2016 Style transfer, neural doodles & texture synthesis 7
A. J. Champandard. “Semantic Style Transfer and Turning Two-Bit Doodles into Fine Artworks”, 2016
Template Mask
Synthesized image User mask
![Page 8: Дмитрий Ульянов, SKOLTECH - MIXAR2016](https://reader034.fdocuments.net/reader034/viewer/2022051007/587fc4ea1a28ab3b158b5cd7/html5/thumbnails/8.jpg)
Dmitry Ulyanov11/13/2016 Style transfer, neural doodles & texture synthesis 8
How does it work?
![Page 9: Дмитрий Ульянов, SKOLTECH - MIXAR2016](https://reader034.fdocuments.net/reader034/viewer/2022051007/587fc4ea1a28ab3b158b5cd7/html5/thumbnails/9.jpg)
Dmitry Ulyanov
Image generation by optimization
11/13/2016 Style transfer, neural doodles & texture synthesis 9
![Page 10: Дмитрий Ульянов, SKOLTECH - MIXAR2016](https://reader034.fdocuments.net/reader034/viewer/2022051007/587fc4ea1a28ab3b158b5cd7/html5/thumbnails/10.jpg)
Dmitry Ulyanov
Texture:
Activations at layer :
Gram matrix at layer :
Gatys et. al.: Optimization-based texture synthesis
11/13/2016 Style transfer, neural doodles & texture synthesis 10
![Page 11: Дмитрий Ульянов, SKOLTECH - MIXAR2016](https://reader034.fdocuments.net/reader034/viewer/2022051007/587fc4ea1a28ab3b158b5cd7/html5/thumbnails/11.jpg)
Dmitry Ulyanov
Image:
Gram matrix at layer :
Loss at layer :
Gatys et. al.: Optimization-based texture synthesis
11/13/2016 Style transfer, neural doodles & texture synthesis 11
![Page 12: Дмитрий Ульянов, SKOLTECH - MIXAR2016](https://reader034.fdocuments.net/reader034/viewer/2022051007/587fc4ea1a28ab3b158b5cd7/html5/thumbnails/12.jpg)
Dmitry Ulyanov
Loss:
Solve
By gradient descent
Gatys et. al.: Optimization-based texture synthesis
11/13/2016 Style transfer, neural doodles & texture synthesis 12
![Page 13: Дмитрий Ульянов, SKOLTECH - MIXAR2016](https://reader034.fdocuments.net/reader034/viewer/2022051007/587fc4ea1a28ab3b158b5cd7/html5/thumbnails/13.jpg)
Dmitry Ulyanov
Examples: Texture Synthesis
11/13/2016 Style transfer, neural doodles & texture synthesis 13
Source Synthesized
L. A. Gatys, A. S. Ecker, M. Bethge; “Texture Synthesis Using Convolutional Neural Networks“; NIPS 2015
![Page 14: Дмитрий Ульянов, SKOLTECH - MIXAR2016](https://reader034.fdocuments.net/reader034/viewer/2022051007/587fc4ea1a28ab3b158b5cd7/html5/thumbnails/14.jpg)
Dmitry Ulyanov
How to: Neural Doodles
11/13/2016 Style transfer, neural doodles & texture synthesis 14
github.com/DmitryUlyanov/fast-neural-doodle
Template Mask
Synthesized image User mask
![Page 15: Дмитрий Ульянов, SKOLTECH - MIXAR2016](https://reader034.fdocuments.net/reader034/viewer/2022051007/587fc4ea1a28ab3b158b5cd7/html5/thumbnails/15.jpg)
Dmitry Ulyanov
Total loss:
Texture loss:
Content loss:
Gatys et. al.: Content loss for style transfer
11/13/2016 Style transfer, neural doodles & texture synthesis 15
Content Style Result
![Page 16: Дмитрий Ульянов, SKOLTECH - MIXAR2016](https://reader034.fdocuments.net/reader034/viewer/2022051007/587fc4ea1a28ab3b158b5cd7/html5/thumbnails/16.jpg)
Dmitry Ulyanov
Content image:
Activations at layer :
Gatys et. al.: Content loss for style transfer
11/13/2016 Style transfer, neural doodles & texture synthesis 16
![Page 17: Дмитрий Ульянов, SKOLTECH - MIXAR2016](https://reader034.fdocuments.net/reader034/viewer/2022051007/587fc4ea1a28ab3b158b5cd7/html5/thumbnails/17.jpg)
Dmitry Ulyanov
Total loss:
Texture loss:
Content loss:
Gatys et. al.: Content loss for style transfer
11/13/2016 Style transfer, neural doodles & texture synthesis 17
Content Style Result
![Page 18: Дмитрий Ульянов, SKOLTECH - MIXAR2016](https://reader034.fdocuments.net/reader034/viewer/2022051007/587fc4ea1a28ab3b158b5cd7/html5/thumbnails/18.jpg)
Dmitry Ulyanov
The results are excellent, but…
It is slow! Several minutes on a high-end GPU.
What else?
11/13/2016 Style transfer, neural doodles & texture synthesis 18
![Page 19: Дмитрий Ульянов, SKOLTECH - MIXAR2016](https://reader034.fdocuments.net/reader034/viewer/2022051007/587fc4ea1a28ab3b158b5cd7/html5/thumbnails/19.jpg)
Texture Networks:Feed-forward Synthesis of Textures and Stylized Images
Dmitry Ulyanov1,2, Vadim Lebedev1,2, Andrea Vedaldi3, Victor Lempitsky2
11/13/2016 Style transfer, neural doodles & texture synthesis 19
ICML 2016
1 2 3
![Page 20: Дмитрий Ульянов, SKOLTECH - MIXAR2016](https://reader034.fdocuments.net/reader034/viewer/2022051007/587fc4ea1a28ab3b158b5cd7/html5/thumbnails/20.jpg)
Dmitry Ulyanov
Instead of solving Solve
Our method: learn a neural net to generate
11/13/2016 Style transfer, neural doodles & texture synthesis 20
• Now
• Generation requires a single evaluation
• But
• Need to make sure does not collapse everything into one point
![Page 21: Дмитрий Ульянов, SKOLTECH - MIXAR2016](https://reader034.fdocuments.net/reader034/viewer/2022051007/587fc4ea1a28ab3b158b5cd7/html5/thumbnails/21.jpg)
Dmitry Ulyanov
Solve
By gradient descent
Generate :
We propose: texture network
11/13/2016 Style transfer, neural doodles & texture synthesis 21
![Page 22: Дмитрий Ульянов, SKOLTECH - MIXAR2016](https://reader034.fdocuments.net/reader034/viewer/2022051007/587fc4ea1a28ab3b158b5cd7/html5/thumbnails/22.jpg)
Dmitry Ulyanov
Solve
By gradient descent
Generate :
We propose: stylization network
11/13/2016 Style transfer, neural doodles & texture synthesis 22
![Page 23: Дмитрий Ульянов, SKOLTECH - MIXAR2016](https://reader034.fdocuments.net/reader034/viewer/2022051007/587fc4ea1a28ab3b158b5cd7/html5/thumbnails/23.jpg)
Dmitry Ulyanov
Qualitative evaluation: textures
11/13/2016 Style transfer, neural doodles & texture synthesis 23
Texture Gatys et. al.
(90 sec.)
Ours
(0.06 sec.)
Almost similar but ours 500 times faster.
![Page 24: Дмитрий Ульянов, SKOLTECH - MIXAR2016](https://reader034.fdocuments.net/reader034/viewer/2022051007/587fc4ea1a28ab3b158b5cd7/html5/thumbnails/24.jpg)
Dmitry Ulyanov
Qualitative evaluation: textures
11/13/2016 Style transfer, neural doodles & texture synthesis 24
Texture Gatys et. al.
(90 sec.)
Ours
(0.06 sec.)
![Page 25: Дмитрий Ульянов, SKOLTECH - MIXAR2016](https://reader034.fdocuments.net/reader034/viewer/2022051007/587fc4ea1a28ab3b158b5cd7/html5/thumbnails/25.jpg)
Dmitry Ulyanov
Qualitative evaluation: textures
11/13/2016 Style transfer, neural doodles & texture synthesis 25
Texture Gatys et. al. Ours Texture Gatys et. al. Ours
(90 sec.) (0.06 sec.) (90 sec.) (0.06 sec.)
![Page 26: Дмитрий Ульянов, SKOLTECH - MIXAR2016](https://reader034.fdocuments.net/reader034/viewer/2022051007/587fc4ea1a28ab3b158b5cd7/html5/thumbnails/26.jpg)
Dmitry Ulyanov
Qualitative results: stylization
11/13/2016 Style transfer, neural doodles & texture synthesis 26
ContentOurs
(0.06 sec.)Style
Gatys et. al.
(90 sec.)
![Page 27: Дмитрий Ульянов, SKOLTECH - MIXAR2016](https://reader034.fdocuments.net/reader034/viewer/2022051007/587fc4ea1a28ab3b158b5cd7/html5/thumbnails/27.jpg)
Dmitry Ulyanov
Qualitative results: stylization
11/13/2016 Style transfer, neural doodles & texture synthesis 27
Content
Ours Gatys et. al.
Style
![Page 28: Дмитрий Ульянов, SKOLTECH - MIXAR2016](https://reader034.fdocuments.net/reader034/viewer/2022051007/587fc4ea1a28ab3b158b5cd7/html5/thumbnails/28.jpg)
Dmitry Ulyanov
Generator network
11/13/2016 Style transfer, neural doodles & texture synthesis 28
• Works good with any fully convolutional architectures.
• Use Instance normalization instead of Batch Normalization.
![Page 29: Дмитрий Ульянов, SKOLTECH - MIXAR2016](https://reader034.fdocuments.net/reader034/viewer/2022051007/587fc4ea1a28ab3b158b5cd7/html5/thumbnails/29.jpg)
Dmitry Ulyanov11/13/2016 Style transfer, neural doodles & texture synthesis 29
Was the technology used somewhere?
Yes!
![Page 30: Дмитрий Ульянов, SKOLTECH - MIXAR2016](https://reader034.fdocuments.net/reader034/viewer/2022051007/587fc4ea1a28ab3b158b5cd7/html5/thumbnails/30.jpg)
Dmitry Ulyanov
Online neural doodles: likemo.net
11/13/2016 Style transfer, neural doodles & texture synthesis 30
GIF: prostheticknowledge-online-neural-doodle
Code: github.com/DmitryUlyanov/online-neural-doodle
![Page 31: Дмитрий Ульянов, SKOLTECH - MIXAR2016](https://reader034.fdocuments.net/reader034/viewer/2022051007/587fc4ea1a28ab3b158b5cd7/html5/thumbnails/31.jpg)
Dmitry Ulyanov
• Made possible many stylization apps for mobile devices
Fast stylization
11/13/2016 Style transfer, neural doodles & texture synthesis 31
![Page 32: Дмитрий Ульянов, SKOLTECH - MIXAR2016](https://reader034.fdocuments.net/reader034/viewer/2022051007/587fc4ea1a28ab3b158b5cd7/html5/thumbnails/32.jpg)
Dmitry Ulyanov
Source code is open at
https://github.com/DmitryUlyanov/
Source code
11/13/2016 Style transfer, neural doodles & texture synthesis 32
![Page 33: Дмитрий Ульянов, SKOLTECH - MIXAR2016](https://reader034.fdocuments.net/reader034/viewer/2022051007/587fc4ea1a28ab3b158b5cd7/html5/thumbnails/33.jpg)
Dmitry Ulyanov
Thank you!
The last slide
11/13/2016 Style transfer, neural doodles & texture synthesis 33
![Page 34: Дмитрий Ульянов, SKOLTECH - MIXAR2016](https://reader034.fdocuments.net/reader034/viewer/2022051007/587fc4ea1a28ab3b158b5cd7/html5/thumbnails/34.jpg)
Dmitry Ulyanov
• Generative Adversarial Networks (Goodfellow et. al., NIPS 2014): a neural
network aims to produce samples that are indistinguishable from real examples
• Perceptual Losses for Real-Time Style Transfer and Super-Resolution, (Johnson
et. al., ECCV 2016): very similar approach fast stylization approach.
• Precomputed Real-Time Texture Synthesis with Markovian Generative
Adversarial Networks (Li & Wand, ECCV 2016): similar patch-based style transfer
acceleration approach.
Related work
11/13/2016 Style transfer, neural doodles & texture synthesis 34
Similar concurrent work
Feed-forward generator