NeuralArt 電腦作畫
-
Upload
mark-chang -
Category
Technology
-
view
10.226 -
download
0
Transcript of NeuralArt 電腦作畫
![Page 1: NeuralArt 電腦作畫](https://reader031.fdocuments.net/reader031/viewer/2022021415/587853711a28ab68198b6c4d/html5/thumbnails/1.jpg)
Neural Art -‐-‐ 電腦作畫 by Mark Chang
![Page 2: NeuralArt 電腦作畫](https://reader031.fdocuments.net/reader031/viewer/2022021415/587853711a28ab68198b6c4d/html5/thumbnails/2.jpg)
A Neural Algorithm of Ar7s7c Style
• 作者: – Leon A. Gatys. – Alexander S. Ecker. – MaAhias Bethge
• 所屬單位: – Werner Reichardt Centre for Integra7ve Neuroscience and Ins7tute of Theore7cal Physics, University of Tubingen, Germany.
– Bernstein Center for Computa7onal Neuroscience, Tubingen, Germany.
![Page 3: NeuralArt 電腦作畫](https://reader031.fdocuments.net/reader031/viewer/2022021415/587853711a28ab68198b6c4d/html5/thumbnails/3.jpg)
如何作畫?
大腦 畫家
景物 畫風 畫作
電腦 類神經網路
![Page 4: NeuralArt 電腦作畫](https://reader031.fdocuments.net/reader031/viewer/2022021415/587853711a28ab68198b6c4d/html5/thumbnails/4.jpg)
大綱
• 人類視覺 • 電腦視覺 • 電腦作畫 • 作品展示
![Page 5: NeuralArt 電腦作畫](https://reader031.fdocuments.net/reader031/viewer/2022021415/587853711a28ab68198b6c4d/html5/thumbnails/5.jpg)
人類視覺
• 神經元 • 視覺傳遞途徑 • 錯覺
![Page 6: NeuralArt 電腦作畫](https://reader031.fdocuments.net/reader031/viewer/2022021415/587853711a28ab68198b6c4d/html5/thumbnails/6.jpg)
神經元
• Neuron • Ac7on Poten7al
Dendrite
Axon
Cell Body
Time
Voltage
Threshold
![Page 7: NeuralArt 電腦作畫](https://reader031.fdocuments.net/reader031/viewer/2022021415/587853711a28ab68198b6c4d/html5/thumbnails/7.jpg)
視覺傳遞途徑
Re7na
Visual Area V1
Visual Area V4 Inferior Temporal Gyrus (IT)
![Page 8: NeuralArt 電腦作畫](https://reader031.fdocuments.net/reader031/viewer/2022021415/587853711a28ab68198b6c4d/html5/thumbnails/8.jpg)
視覺傳遞途徑
Visual Area V1
Inferior Temporal Gyrus (IT)
Recep7ve Fields
Visual Area V4
![Page 9: NeuralArt 電腦作畫](https://reader031.fdocuments.net/reader031/viewer/2022021415/587853711a28ab68198b6c4d/html5/thumbnails/9.jpg)
錯覺
![Page 10: NeuralArt 電腦作畫](https://reader031.fdocuments.net/reader031/viewer/2022021415/587853711a28ab68198b6c4d/html5/thumbnails/10.jpg)
電腦視覺
• Neural Networks • Convolu7onal Neural Networks • VGG 19
![Page 11: NeuralArt 電腦作畫](https://reader031.fdocuments.net/reader031/viewer/2022021415/587853711a28ab68198b6c4d/html5/thumbnails/11.jpg)
Neural Networks
n
W1
W2
x1
x2
b Wb
y
nin = w1x1 + w2x2 + wb
nout
=1
1 + e�nin
Sigmoid
Rec7fied Linear
nout
=
⇢nin
if nin
> 0
0 otherwise
![Page 12: NeuralArt 電腦作畫](https://reader031.fdocuments.net/reader031/viewer/2022021415/587853711a28ab68198b6c4d/html5/thumbnails/12.jpg)
Neural Networks
x
y
n11
n12
n21
n22
b b
z1
z2
Input Layer
Hidden Layer
Output Layer
W12,y
W12,x
W11,y
W11,b W12,b
W11,x W21,11
W22,12
W21,12
W22,11
W21,b W22,b
![Page 13: NeuralArt 電腦作畫](https://reader031.fdocuments.net/reader031/viewer/2022021415/587853711a28ab68198b6c4d/html5/thumbnails/13.jpg)
Convolu7onal Neural Networks
• Convolu7onal Layer
depth
width width depth
weights weights
height
shared weight
![Page 14: NeuralArt 電腦作畫](https://reader031.fdocuments.net/reader031/viewer/2022021415/587853711a28ab68198b6c4d/html5/thumbnails/14.jpg)
Convolu7onal Neural Networks
• Stride • Padding
Stride = 1
Stride = 2
Padding = 0
Padding = 1
![Page 15: NeuralArt 電腦作畫](https://reader031.fdocuments.net/reader031/viewer/2022021415/587853711a28ab68198b6c4d/html5/thumbnails/15.jpg)
Convolu7onal Neural Networks
• Pooling Layer 1 3 2 4
5 7 6 8
0 0 4 4
6 6 0 0
4 5
3 2
no overlap
no padding no weights
depth = 1
7 8
6 4
Maximum Pooling
Average Pooling
![Page 16: NeuralArt 電腦作畫](https://reader031.fdocuments.net/reader031/viewer/2022021415/587853711a28ab68198b6c4d/html5/thumbnails/16.jpg)
Convolu7onal Neural Networks
Convolu7onal Layer
Convolu7onal Layer Pooling
Layer
Pooling Layer
Recep7ve Fields Recep7ve Fields
Input Layer
![Page 17: NeuralArt 電腦作畫](https://reader031.fdocuments.net/reader031/viewer/2022021415/587853711a28ab68198b6c4d/html5/thumbnails/17.jpg)
Convolu7onal Neural Networks
Input Layer
Convolu7onal Layer with
Recep7ve Fields:
Max-‐pooling Layer with
Width =3, Height = 3
Filter Responses
Filter Responses
Input Image
![Page 18: NeuralArt 電腦作畫](https://reader031.fdocuments.net/reader031/viewer/2022021415/587853711a28ab68198b6c4d/html5/thumbnails/18.jpg)
VGG 19
• Karen Simonyan & Andrew Zisserman. Very Deep Convolu7onal Networks for Large-‐scale Image Recogni7on.
• ImageNet Challenge 2014 • 19 (+5) layers – 16 Convolu7onal layers (width=3, height=3) – 5 Max-‐pooling layers (width=2, height=2) – 3 Fully-‐connected layers
![Page 19: NeuralArt 電腦作畫](https://reader031.fdocuments.net/reader031/viewer/2022021415/587853711a28ab68198b6c4d/html5/thumbnails/19.jpg)
VGG 19
depth=64 conv1_1 conv1_2
maxpool
depth=128 conv2_1 conv2_2
maxpool
depth=256 conv3_1 conv3_2 conv3_3 conv3_4
depth=512 conv4_1 conv4_2 conv4_3 conv4_4
depth=512 conv5_1 conv5_2 conv5_3 conv5_4
maxpool maxpool maxpool
size=4096 FC1 FC2
size=1000 sogmax
![Page 20: NeuralArt 電腦作畫](https://reader031.fdocuments.net/reader031/viewer/2022021415/587853711a28ab68198b6c4d/html5/thumbnails/20.jpg)
電腦作畫
• 內容產生 • 畫風產生 • 作品產生
![Page 21: NeuralArt 電腦作畫](https://reader031.fdocuments.net/reader031/viewer/2022021415/587853711a28ab68198b6c4d/html5/thumbnails/21.jpg)
內容產生
大腦 畫家 景物
畫布 計算 兩者 差異
神經反應
補上線條和顏色
![Page 22: NeuralArt 電腦作畫](https://reader031.fdocuments.net/reader031/viewer/2022021415/587853711a28ab68198b6c4d/html5/thumbnails/22.jpg)
內容產生
計算 兩者 差異
Filter Responses VGG19
修正差異
相片
畫布
修正後
Width*Height Depth
![Page 23: NeuralArt 電腦作畫](https://reader031.fdocuments.net/reader031/viewer/2022021415/587853711a28ab68198b6c4d/html5/thumbnails/23.jpg)
內容產生
Layer l’s Filter l Responses:
Lcontent
(P l, Cl) =1
2
X
i,j
(Cl
i,j
� P l
i,j
)2
Layer l’s Filter Responses: Lcontent
(P l, Cl) =1
2
X
i,j
(Cl
i,j
� P l
i,j
)2Input Photo: L
content
(p, c, l) =1
2
X
i,j
(Cl
i,j
� P l
i,j
)2
Lcontent
(p,x, l) =1
2
X
i,j
(X l
i,j
� P l
i,j
)2
@Lcontent
(p,x, l)
@X l
i,j
= X l
i,j
� P l
i,j
X l
X li,j
Input Canvas: x
Width*Height (j)
Depth (i)
Width*Height (j)
Depth (i)
![Page 24: NeuralArt 電腦作畫](https://reader031.fdocuments.net/reader031/viewer/2022021415/587853711a28ab68198b6c4d/html5/thumbnails/24.jpg)
內容產生
• Backward Propaga7on Layer l’s Filter l Responses: X l
Input Canvas:
x
VGG19
@Lcontent
@x=
@Lcontent
@X l
@X l
x
x x� ⌘@L
content
@xUpdate Canvas
Learning Rate
![Page 25: NeuralArt 電腦作畫](https://reader031.fdocuments.net/reader031/viewer/2022021415/587853711a28ab68198b6c4d/html5/thumbnails/25.jpg)
內容產生
![Page 26: NeuralArt 電腦作畫](https://reader031.fdocuments.net/reader031/viewer/2022021415/587853711a28ab68198b6c4d/html5/thumbnails/26.jpg)
內容產生
VGG19
conv1_2 conv2_2 conv3_4 conv4_4 conv5_2 conv5_1
![Page 27: NeuralArt 電腦作畫](https://reader031.fdocuments.net/reader031/viewer/2022021415/587853711a28ab68198b6c4d/html5/thumbnails/27.jpg)
畫風產生
• ”Style” is posi7on-‐independent
style extrac7on
![Page 28: NeuralArt 電腦作畫](https://reader031.fdocuments.net/reader031/viewer/2022021415/587853711a28ab68198b6c4d/html5/thumbnails/28.jpg)
畫風產生
VGG19 畫作
G
G
Filter Responses Gram Matrix
Width*Height
Depth
Depth
Depth
Posi7on-‐ dependent
Posi7on-‐ independent
![Page 29: NeuralArt 電腦作畫](https://reader031.fdocuments.net/reader031/viewer/2022021415/587853711a28ab68198b6c4d/html5/thumbnails/29.jpg)
畫風產生
1. .5
.5
.5
1.
1. .5 .25 1.
.5 .25 .5
.25 .25
1. .5 1.
Width*Height
Depth
k1 k2
k1
k2
Depth
Depth
Layer l’s Filter Responses Gram Matrix
F l1
F l2
F l3
F l4
F l1
F l2
F l3
F l4
G
Gli,j = F l
i · F lj
Gl4,1 = F l
4 · F l1
= 1⇥ 1 + 0⇥ 0.5 + 0⇥ 0 + ...
= 1
![Page 30: NeuralArt 電腦作畫](https://reader031.fdocuments.net/reader031/viewer/2022021415/587853711a28ab68198b6c4d/html5/thumbnails/30.jpg)
畫風產生
VGG19 Filter
Responses Gram Matrix
計算 兩者 差異
G
G
畫風
畫布
修正差異
修正後
![Page 31: NeuralArt 電腦作畫](https://reader031.fdocuments.net/reader031/viewer/2022021415/587853711a28ab68198b6c4d/html5/thumbnails/31.jpg)
畫風產生
Lstyle(a,x, l) =1
2
X
i,j
(X li,j �Al
i,j)2
@Lstyle(a,x, l)
@F li,j
= ((F l)T (X l �Al))j,i
Layer l’s Filter Responses
Layer l’s Gram Matrix
Layer l’s Gram Matrix
F li,j
Ali,j X l
i,j
Input Artwork:
Input Canvas: a
x
![Page 32: NeuralArt 電腦作畫](https://reader031.fdocuments.net/reader031/viewer/2022021415/587853711a28ab68198b6c4d/html5/thumbnails/32.jpg)
畫風產生
![Page 33: NeuralArt 電腦作畫](https://reader031.fdocuments.net/reader031/viewer/2022021415/587853711a28ab68198b6c4d/html5/thumbnails/33.jpg)
畫風產生
VGG19
Conv1_1 Conv1_1 Conv2_1
Conv1_1 Conv2_1 Conv3_1
Conv1_1 Conv2_1 Conv3_1 Conv4_1
Conv1_1 Conv2_1 Conv3_1 Conv4_1 Conv5_1
![Page 34: NeuralArt 電腦作畫](https://reader031.fdocuments.net/reader031/viewer/2022021415/587853711a28ab68198b6c4d/html5/thumbnails/34.jpg)
作品產生
Ltotal
= ↵Lcontent
+ �Lstyle
a
Lcontent
(p, c, l) =1
2
X
i,j
(Cl
i,j
� P l
i,j
)2
x
x x� ⌘@L
total
@xx
Filter Responses VGG19
Lcontent
(p,x)
Lstyle(a,x)
Gram Matrix
![Page 35: NeuralArt 電腦作畫](https://reader031.fdocuments.net/reader031/viewer/2022021415/587853711a28ab68198b6c4d/html5/thumbnails/35.jpg)
作品產生
VGG19 VGG19
Lcontent
(p,x) Lstyle(a,x)
Conv1_1 Conv2_1 Conv3_1 Conv4_1 Conv5_1
Conv4_2
Ltotal
= ↵Lcontent
+ �Lstyle
![Page 36: NeuralArt 電腦作畫](https://reader031.fdocuments.net/reader031/viewer/2022021415/587853711a28ab68198b6c4d/html5/thumbnails/36.jpg)
作品產生
![Page 37: NeuralArt 電腦作畫](https://reader031.fdocuments.net/reader031/viewer/2022021415/587853711a28ab68198b6c4d/html5/thumbnails/37.jpg)
作品展示
• 內容 v.s. 畫風 • 不同起始狀態 • 不同VGG Layers • 素描、水彩 • 詩中有畫、畫中有詩
![Page 38: NeuralArt 電腦作畫](https://reader031.fdocuments.net/reader031/viewer/2022021415/587853711a28ab68198b6c4d/html5/thumbnails/38.jpg)
內容 v.s. 畫風
0.15 0.05
0.02 0.007
↵
�
![Page 39: NeuralArt 電腦作畫](https://reader031.fdocuments.net/reader031/viewer/2022021415/587853711a28ab68198b6c4d/html5/thumbnails/39.jpg)
不同起始狀態 noise 0.9 *noise + 0.1*photo photo
![Page 40: NeuralArt 電腦作畫](https://reader031.fdocuments.net/reader031/viewer/2022021415/587853711a28ab68198b6c4d/html5/thumbnails/40.jpg)
不同VGG Layers
Conv1_1 Conv2_1
Conv1_1 Conv2_1 Conv3_1
Conv1_1 Conv2_1 Conv3_1 Conv4_1
Conv1_1 Conv2_1 Conv3_1 Conv4_1 Conv5_1
↵
�= 0.002
![Page 41: NeuralArt 電腦作畫](https://reader031.fdocuments.net/reader031/viewer/2022021415/587853711a28ab68198b6c4d/html5/thumbnails/41.jpg)
素描、水彩
![Page 42: NeuralArt 電腦作畫](https://reader031.fdocuments.net/reader031/viewer/2022021415/587853711a28ab68198b6c4d/html5/thumbnails/42.jpg)
詩中有畫、畫中有詩
![Page 43: NeuralArt 電腦作畫](https://reader031.fdocuments.net/reader031/viewer/2022021415/587853711a28ab68198b6c4d/html5/thumbnails/43.jpg)
延伸閱讀 • A Neural Algorithm of Ar7s7c Style – hAp://arxiv.org/abs/1508.06576
• Texture Synthesis Using Convolu7onal Neural Networks – hAp://arxiv.org/abs/1505.07376
• Convolu7onal Neural Network – hAp://cs231n.github.io/convolu7onal-‐networks/
• Neural Network Back Propaga7on – hAp://cpmarkchang.logdown.com/posts/277349-‐neural-‐network-‐backward-‐propaga7on
• 電腦賦詩: – hAp://www.slideshare.net/ckmarkohchang/computa7onal-‐poetry
![Page 44: NeuralArt 電腦作畫](https://reader031.fdocuments.net/reader031/viewer/2022021415/587853711a28ab68198b6c4d/html5/thumbnails/44.jpg)
程式碼
• Python Tensorflow – hAps://github.com/ckmarkoh/neuralart_tensorflow
• Python Theano – hAps://github.com/woonketwong/ar7fy
• Python Theano (ipython notebook) – hAps://github.com/Lasagne/Recipes/blob/master/examples/styletransfer/Art%20Style%20Transfer.ipynb
• Python deeppy – hAps://github.com/andersbll/neural_ar7s7c_style
![Page 45: NeuralArt 電腦作畫](https://reader031.fdocuments.net/reader031/viewer/2022021415/587853711a28ab68198b6c4d/html5/thumbnails/45.jpg)
圖片來源
• hAp://www.taipei-‐101.com.tw/upload/news/201502/2015021711505431705145.JPG
• hAps://github.com/andersbll/neural_ar7s7c_style/blob/master/images/starry_night.jpg?raw=true
![Page 46: NeuralArt 電腦作畫](https://reader031.fdocuments.net/reader031/viewer/2022021415/587853711a28ab68198b6c4d/html5/thumbnails/46.jpg)
特別感謝
• 臺大資工imlab