A Neural Algorithm of Artistic Style

Liyuan Su Elaheh YousefiAmiri

February 1, 2019

Liyuan Su, Elaheh YousefiAmiri A Neural Algorithm of Artistic Style February 1, 2019 1 / 16

Overview

1 Problem Statement

2 Methods of Artistic Styles

3 Deep image representations

Liyuan Su, Elaheh YousefiAmiri A Neural Algorithm of Artistic Style February 1, 2019 2 / 16

Problem Statement

The generated image B combines the ”content” of the image A with the”style” of image S.

Liyuan Su, Elaheh YousefiAmiri A Neural Algorithm of Artistic Style February 1, 2019 3 / 16

Methods of Artistic StylesImage Style Transfer

Traditional Methods: Non-parametric

Deep Learning based Methods

� Optimization� Convolutional Neural Networks(Gatys et al.)� Feed-forward(Johnson et al.)

Liyuan Su, Elaheh YousefiAmiri A Neural Algorithm of Artistic Style February 1, 2019 4 / 16

Methods of Artistic StylesOptimization Method

Optimization

”Neural style transfer used an optimization technique that is, starting offwith a random noise image and making it more and more desirable withevery training iteration of the neural network.”

Liyuan Su, Elaheh YousefiAmiri A Neural Algorithm of Artistic Style February 1, 2019 5 / 16

Methods of Artistic StylesNeural Style Transfer Algorithm-CNN

Figure: Style and content representations taken at each layer of a NeuralNetwork.Image from(Gatys et)

Liyuan Su, Elaheh YousefiAmiri A Neural Algorithm of Artistic Style February 1, 2019 6 / 16

Methods of Artistic StylesFeed-Forward

Feed-Forward

”By pre-training a feed-forward network rather than directly optimizing theloss functions.”

Liyuan Su, Elaheh YousefiAmiri A Neural Algorithm of Artistic Style February 1, 2019 7 / 16

Methods of Artistic StylesFeed-Forward

Figure: ”System overview. We train an image transformation network totransform input images into output images. We use a loss network pretrained forimage classification to define perceptual loss functions that measure perceptualdifferences in content and style between images. The loss network remains fixedduring the training process.” Figure from Johnson et al

Liyuan Su, Elaheh YousefiAmiri A Neural Algorithm of Artistic Style February 1, 2019 8 / 16

Deep image representations

Content representation

Style representation

Style transfer

Liyuan Su, Elaheh YousefiAmiri A Neural Algorithm of Artistic Style February 1, 2019 9 / 16

Deep image representationsContent representations

Define the squared-error loss between the two feature representations

Lcontent(~p, ~x , l) =1

2

∑i ,j

(F lij − P l

ij)2

The derivative of this loss with respect to the activations in layer l equal

∂Lcontent

∂F lij

=

{(F l − P l)ij F l

ij > 0,

0 otherwise

from which the gradient with respect to the image ~x can be computedusing standard error back-propagation.

Liyuan Su, Elaheh YousefiAmiri A Neural Algorithm of Artistic Style February 1, 2019 10 / 16

Deep image representationsstyle representation

Feature correlations are given by the Gram matrix G l ∈ RNl×Nl ,where G lij

is the inner product between the vectorised feature maps i and j in layer l:

G lij =

∑k

F likF

ljk .

The contribution of layer l to the total loss is then:

El =1

4N2l M

2l

∑i ,j

(G lij − Al

ij)2

and the total style loss is

Lstyle(~a, ~x) =L∑

l=0

wlEl

where wl are weighting factors of the contribution of each layer to thetotal loss

Liyuan Su, Elaheh YousefiAmiri A Neural Algorithm of Artistic Style February 1, 2019 11 / 16

Deep image representationsStyle representation

The derivative of El with respect to the activations in layer l can becomputed analytically:

∂El

∂F lij

=

{1

N2l M

2l

((F l)T (G l − Al))ji if F lij > 0,

0 otherwise

The gradients of El with respect to the pixel values ~x can be readilycomputed using standard error back-propagation.

Liyuan Su, Elaheh YousefiAmiri A Neural Algorithm of Artistic Style February 1, 2019 12 / 16

Deep image representationsStyle transfer

The loss function we minimise is

Ltotal(~p, ~a, ~x) = αLcontent(~p, ~x) + βLstyle(~a, ~x)

Figure: Style transfer algorithm

Liyuan Su, Elaheh YousefiAmiri A Neural Algorithm of Artistic Style February 1, 2019 13 / 16

online Aplications

Style transfer Apps:http://deepart.iohttp://www.pikazoapp.comhttps://artisto.my.com (Video and Photo Editor)

Liyuan Su, Elaheh YousefiAmiri A Neural Algorithm of Artistic Style February 1, 2019 14 / 16

References

Leon A. Gatys, Alexander S. Ecker, Matthias Bethge(2015)

A Neural Algorithm of Artistic Style

https://arxiv.org/abs/1508.06576 .

Justin Johnson, Alexandre Alahi, Li Fei-Fei(2016)

Perceptual Losses for Real-Time Style Transfer and Super-Resolution

https://arxiv.org/pdf/1603.08155.pdf

Liyuan Su, Elaheh YousefiAmiri A Neural Algorithm of Artistic Style February 1, 2019 15 / 16

Thank YouThe End

Liyuan Su, Elaheh YousefiAmiri A Neural Algorithm of Artistic Style February 1, 2019 16 / 16