Lecture 6: Training Neural Networks, Part I -...

Fei-Fei Li & Justin Johnson & Serena Yeung Lecture 6 - April 20, 2017Fei-Fei Li & Justin Johnson & Serena Yeung Lecture 6 - April 20, 20171

Lecture 6:Training Neural Networks,

Part I


Administrative

Assignment 1 due Thursday (today), 11:59pm on Canvas

Assignment 2 out today

Project proposal due Tuesday April 25

Notes on backprop for a linear layer and vector/tensor derivatives linked to Lecture 4 on syllabus

Fei-Fei Li & Justin Johnson & Serena Yeung Lecture 6 - April 20, 2017

Where we are now...

3

x

W

hinge loss

R

+ Ls (scores)

Computational graphs

*


Where we are now...

Linear score function:

2-layer Neural Network

x hW1 sW2

3072 100 10

Neural Networks


Illustration of LeCun et al. 1998 from CS231n 2017 Lecture 1

5

Where we are now...

Convolutional Neural Networks


Where we are now...Convolutional Layer

32

32

3

32x32x3 image5x5x3 filter

convolve (slide) over all spatial locations

activation map

1

28

28


Where we are now...Convolutional Layer

32

32

3

Convolution Layer

activation maps

6

28

28

For example, if we had 6 5x5 filters, we’ll get 6 separate activation maps:

We stack these up to get a “new image” of size 28x28x6!


Where we are now...

Landscape image is CC0 1.0 public domainWalking man image is CC0 1.0 public domain

Learning network parameters through optimization

http://maxpixel.freegreatpicture.com/Mountains-Valleys-Landscape-Hills-Grass-Green-699369

https://creativecommons.org/publicdomain/zero/1.0/

http://maxpixel.freegreatpicture.com/Mountains-Valleys-Landscape-Hills-Grass-Green-699369

http://www.publicdomainpictures.net/view-image.php?image=139314&picture=walking-man

https://creativecommons.org/publicdomain/zero/1.0/

http://www.publicdomainpictures.net/view-image.php?image=139314&picture=walking-man


Where we are now...

Mini-batch SGDLoop:1. Sample a batch of data2. Forward prop it through the graph

(network), get loss3. Backprop to calculate the gradients4. Update the parameters using the gradient


Next: Training Neural Networks


Overview1. One time setup

activation functions, preprocessing, weight initialization, regularization, gradient checking

2. Training dynamicsbabysitting the learning process, parameter updates, hyperparameter optimization

3. Evaluationmodel ensembles


Part 1- Activation Functions- Data Preprocessing- Weight Initialization- Batch Normalization- Babysitting the Learning Process- Hyperparameter Optimization


Activation Functions


Activation FunctionsSigmoid

tanh

ReLU

Leaky ReLU

Maxout

ELU



Sigmoid

- Squashes numbers to range [0,1]- Historically popular since they

have nice interpretation as a saturating “firing rate” of a neuron



Sigmoid



3 problems:

1. Saturated neurons “kill” the gradients


sigmoid gate

x

What happens when x = -10?What happens when x = 0?What happens when x = 10?



Sigmoid



3 problems:


2. Sigmoid outputs are not zero-centered


Consider what happens when the input to a neuron (x) is always positive:

What can we say about the gradients on w?


Consider what happens when the input to a neuron is always positive...

What can we say about the gradients on w?Always all positive or all negative :((this is also why you want zero-mean data!)

hypothetical optimal w vector

zig zag path

allowed gradient update directions




Sigmoid



3 problems:


2. Sigmoid outputs are not zero-centered

3. exp() is a bit compute expensive



tanh(x)

- Squashes numbers to range [-1,1]- zero centered (nice)- still kills gradients when saturated :(

[LeCun et al., 1991]


Activation Functions - Computes f(x) = max(0,x)

- Does not saturate (in +region)- Very computationally efficient- Converges much faster than

sigmoid/tanh in practice (e.g. 6x)- Actually more biologically plausible

than sigmoid

ReLU(Rectified Linear Unit)

[Krizhevsky et al., 2012]



ReLU(Rectified Linear Unit)

- Computes f(x) = max(0,x)

- Does not saturate (in +region)- Very computationally efficient- Converges much faster than

sigmoid/tanh in practice (e.g. 6x)- Actually more biologically plausible

than sigmoid

- Not zero-centered output- An annoyance:

hint: what is the gradient when x < 0?


ReLU gate

x

What happens when x = -10?What happens when x = 0?What happens when x = 10?


DATA CLOUDactive ReLU

dead ReLUwill never activate => never update


DATA CLOUDactive ReLU

dead ReLUwill never activate => never update

=> people like to initialize ReLU neurons with slightly positive biases (e.g. 0.01)



Leaky ReLU

- Does not saturate- Computationally efficient- Converges much faster than

sigmoid/tanh in practice! (e.g. 6x)- will not “die”.

[Mass et al., 2013][He et al., 2015]



Leaky ReLU

- Does not saturate- Computationally efficient- Converges much faster than

sigmoid/tanh in practice! (e.g. 6x)- will not “die”.

Parametric Rectifier (PReLU)

backprop into \alpha(parameter)

[Mass et al., 2013][He et al., 2015]


Activation FunctionsExponential Linear Units (ELU)

- All benefits of ReLU- Closer to zero mean outputs- Negative saturation regime

compared with Leaky ReLU adds some robustness to noise

- Computation requires exp()

[Clevert et al., 2015]


Maxout “Neuron”- Does not have the basic form of dot product ->

nonlinearity- Generalizes ReLU and Leaky ReLU - Linear Regime! Does not saturate! Does not die!

Problem: doubles the number of parameters/neuron :(

[Goodfellow et al., 2013]


TLDR: In practice:

- Use ReLU. Be careful with your learning rates- Try out Leaky ReLU / Maxout / ELU- Try out tanh but don’t expect much- Don’t use sigmoid


Data Preprocessing


Step 1: Preprocess the data

(Assume X [NxD] is data matrix, each example in a row)


Remember: Consider what happens when the input to a neuron is always positive...

What can we say about the gradients on w?Always all positive or all negative :((this is also why you want zero-mean data!)

hypothetical optimal w vector

zig zag path




Step 1: Preprocess the dataIn practice, you may also see PCA and Whitening of the data

(data has diagonal covariance matrix)

(covariance matrix is the identity matrix)


TLDR: In practice for Images: center only

- Subtract the mean image (e.g. AlexNet)(mean image = [32,32,3] array)

- Subtract per-channel mean (e.g. VGGNet)(mean along each channel = 3 numbers)

e.g. consider CIFAR-10 example with [32,32,3] images

Not common to normalize variance, to do PCA or whitening


Weight Initialization


- Q: what happens when W=0 init is used?


- First idea: Small random numbers (gaussian with zero mean and 1e-2 standard deviation)


- First idea: Small random numbers (gaussian with zero mean and 1e-2 standard deviation)

Works ~okay for small networks, but problems with deeper networks.


Lets look at some activation statistics

E.g. 10-layer net with 500 neurons on each layer, using tanh non-linearities, and initializing as described in last slide.


All activations become zero!

Q: think about the backward pass. What do the gradients look like?

Hint: think about backward pass for a W*X gate.


Almost all neurons completely saturated, either -1 and 1. Gradients will be all zero.

*1.0 instead of *0.01


“Xavier initialization”[Glorot et al., 2010]

Reasonable initialization.(Mathematical derivation assumes linear activations)


but when using the ReLU nonlinearity it breaks.


He et al., 2015(note additional /2)


Proper initialization is an active area of research…Understanding the difficulty of training deep feedforward neural networksby Glorot and Bengio, 2010

Exact solutions to the nonlinear dynamics of learning in deep linear neural networks by Saxe et al, 2013

Random walk initialization for training very deep feedforward networks by Sussillo and Abbott, 2014

Delving deep into rectifiers: Surpassing human-level performance on ImageNet classification by He et al., 2015

Data-dependent Initializations of Convolutional Neural Networks by Krähenbühl et al., 2015

All you need is a good init, Mishkin and Matas, 2015…


Batch Normalization


Batch Normalization“you want unit gaussian activations? just make them so.”

[Ioffe and Szegedy, 2015]

consider a batch of activations at some layer. To make each dimension unit gaussian, apply:

this is a vanilla differentiable function...


Batch Normalization“you want unit gaussian activations? just make them so.”

[Ioffe and Szegedy, 2015]

XN

D

1. compute the empirical mean and variance independently for each dimension.

2. Normalize


Batch Normalization [Ioffe and Szegedy, 2015]

FC

BN

tanh

FC

BN

tanh

...

Usually inserted after Fully Connected or Convolutional layers, and before nonlinearity.



FC

BN

tanh

FC

BN

tanh

...

Usually inserted after Fully Connected or Convolutional layers, and before nonlinearity.

Problem: do we necessarily want a unit gaussian input to a tanh layer?



And then allow the network to squash the range if it wants to:

Note, the network can learn:

to recover the identity mapping.

Normalize:



- Improves gradient flow through the network

- Allows higher learning rates- Reduces the strong dependence

on initialization- Acts as a form of regularization

in a funny way, and slightly reduces the need for dropout, maybe



Note: at test time BatchNorm layer functions differently:

The mean/std are not computed based on the batch. Instead, a single fixed empirical mean of activations during training is used.

(e.g. can be estimated during training with running averages)


Babysitting the Learning Process


Step 2: Choose the architecture:say we start with one hidden layer of 50 neurons:

input layer hidden layer

output layerCIFAR-10 images, 3072 numbers

10 output neurons, one per class

50 hidden neurons


Double check that the loss is reasonable:

returns the loss and the gradient for all parameters

disable regularization

loss ~2.3.“correct “ for 10 classes


Double check that the loss is reasonable:

crank up regularization

loss went up, good. (sanity check)


Lets try to train now…

Tip: Make sure that you can overfit very small portion of the training data The above code:

- take the first 20 examples from CIFAR-10

- turn off regularization (reg = 0.0)- use simple vanilla ‘sgd’



Tip: Make sure that you can overfit very small portion of the training data

Very small loss, train accuracy 1.00, nice!



Start with small regularization and find learning rate that makes the loss go down.




Loss barely changing




loss not going down:learning rate too low

Loss barely changing: Learning rate is probably too low





Loss barely changing: Learning rate is probably too low

Notice train/val accuracy goes to 20% though, what’s up with that? (remember this is softmax)





Now let’s try learning rate 1e6.


cost: NaN almost always means high learning rate...



loss not going down:learning rate too lowloss exploding:learning rate too high




loss not going down:learning rate too lowloss exploding:learning rate too high

3e-3 is still too high. Cost explodes….

=> Rough range for learning rate we should be cross-validating is somewhere [1e-3 … 1e-5]


Hyperparameter Optimization


Cross-validation strategycoarse -> fine cross-validation in stages

First stage: only a few epochs to get rough idea of what params workSecond stage: longer running time, finer search… (repeat as necessary)

Tip for detecting explosions in the solver: If the cost is ever > 3 * original cost, break out early


For example: run coarse search for 5 epochs

nice

note it’s best to optimize in log space!


Now run finer search...adjust range

53% - relatively good for a 2-layer neural net with 50 hidden neurons.


Now run finer search...adjust range

53% - relatively good for a 2-layer neural net with 50 hidden neurons.

But this best cross-validation result is worrying. Why?


Random Search vs. Grid Search

Important Parameter Important ParameterU

nim

porta

nt P

aram

eter

Uni

mpo

rtant

Par

amet

er

Grid Layout Random Layout

Illustration of Bergstra et al., 2012 by Shayne Longpre, copyright CS231n 2017

Random Search for Hyper-Parameter OptimizationBergstra and Bengio, 2012


Hyperparameters to play with:- network architecture- learning rate, its decay schedule, update type- regularization (L2/Dropout strength)

neural networks practitionermusic = loss function

This image by Paolo Guereta is licensed under CC-BY 2.0

https://commons.wikimedia.org/wiki/File:Pioneer_DJ_equipment_-_angled_left_-_Expomusic_2014.jpg

https://creativecommons.org/licenses/by/3.0/us/

https://commons.wikimedia.org/wiki/File:Pioneer_DJ_equipment_-_angled_left_-_Expomusic_2014.jpg


Cross-validation “command center”


Monitor and visualize the loss curve


Loss

time


Loss

time

Bad initializationa prime suspect


Monitor and visualize the accuracy:

big gap = overfitting=> increase regularization strength?

no gap=> increase model capacity?


Track the ratio of weight updates / weight magnitudes:

ratio between the updates and values: ~ 0.0002 / 0.02 = 0.01 (about okay)want this to be somewhere around 0.001 or so


SummaryWe looked in detail at:

- Activation Functions (use ReLU)- Data Preprocessing (images: subtract mean)- Weight Initialization (use Xavier init)- Batch Normalization (use)- Babysitting the Learning process- Hyperparameter Optimization

(random sample hyperparams, in log space when appropriate)

TLDRs


Next time: Training Neural Networks, Part 2

- Parameter update schemes- Learning rate schedules- Gradient checking- Regularization (Dropout etc.)- Evaluation (Ensembles etc.)- Transfer learning / fine-tuning

Lecture 6: Training Neural Networks, Part I -...

Documents

Transcript of Lecture 6: Training Neural Networks, Part I -...