Keras: An Introduction - University of Waterloo...Dylan Drover STAT 946 Keras: An Introduction....
Transcript of Keras: An Introduction - University of Waterloo...Dylan Drover STAT 946 Keras: An Introduction....
![Page 1: Keras: An Introduction - University of Waterloo...Dylan Drover STAT 946 Keras: An Introduction. General Design General idea is to based on layers and their input/output Prepare your](https://reader034.fdocuments.net/reader034/viewer/2022042119/5e9858e37a49a1288276752a/html5/thumbnails/1.jpg)
Keras: An Introduction
Dylan Drover
STAT 946
December 2, 2015
Dylan Drover STAT 946
Keras: An Introduction
![Page 2: Keras: An Introduction - University of Waterloo...Dylan Drover STAT 946 Keras: An Introduction. General Design General idea is to based on layers and their input/output Prepare your](https://reader034.fdocuments.net/reader034/viewer/2022042119/5e9858e37a49a1288276752a/html5/thumbnails/2.jpg)
Overview
What is Keras?
Neural Network library written in Python
Designed to be minimalistic & straight forward yet extensive
Built on top of either Theano as newly TensorFlow
Why use Keras?
Simple to get started, simple to keep going
Written in python and highly modular; easy to expand
Deep enough to build serious models
Dylan Drover STAT 946
Keras: An Introduction
![Page 3: Keras: An Introduction - University of Waterloo...Dylan Drover STAT 946 Keras: An Introduction. General Design General idea is to based on layers and their input/output Prepare your](https://reader034.fdocuments.net/reader034/viewer/2022042119/5e9858e37a49a1288276752a/html5/thumbnails/3.jpg)
General Design
General idea is to based on layers and their input/output
Prepare your inputs and output tensors
Create first layer to handle input tensor
Create output layer to handle targets
Build virtually any model you like in between
Dylan Drover STAT 946
Keras: An Introduction
![Page 4: Keras: An Introduction - University of Waterloo...Dylan Drover STAT 946 Keras: An Introduction. General Design General idea is to based on layers and their input/output Prepare your](https://reader034.fdocuments.net/reader034/viewer/2022042119/5e9858e37a49a1288276752a/html5/thumbnails/4.jpg)
Layers and Layers (like an Ogre)
Keras has a number of pre-built layers. Notable examples include:
Regular dense, MLP type
Recurrent layers, LSTM, GRU, etc.
Dylan Drover STAT 946
Keras: An Introduction
![Page 5: Keras: An Introduction - University of Waterloo...Dylan Drover STAT 946 Keras: An Introduction. General Design General idea is to based on layers and their input/output Prepare your](https://reader034.fdocuments.net/reader034/viewer/2022042119/5e9858e37a49a1288276752a/html5/thumbnails/5.jpg)
1D Convolutional layers
2D Convolutional layers
Dylan Drover STAT 946
Keras: An Introduction
![Page 6: Keras: An Introduction - University of Waterloo...Dylan Drover STAT 946 Keras: An Introduction. General Design General idea is to based on layers and their input/output Prepare your](https://reader034.fdocuments.net/reader034/viewer/2022042119/5e9858e37a49a1288276752a/html5/thumbnails/6.jpg)
Autoencoders can be built with any other type of layer
Dylan Drover STAT 946
Keras: An Introduction
![Page 7: Keras: An Introduction - University of Waterloo...Dylan Drover STAT 946 Keras: An Introduction. General Design General idea is to based on layers and their input/output Prepare your](https://reader034.fdocuments.net/reader034/viewer/2022042119/5e9858e37a49a1288276752a/html5/thumbnails/7.jpg)
Other types of layer include:
Dropout
Noise
Pooling
Normalization
Embedding
And many more...
Dylan Drover STAT 946
Keras: An Introduction
![Page 8: Keras: An Introduction - University of Waterloo...Dylan Drover STAT 946 Keras: An Introduction. General Design General idea is to based on layers and their input/output Prepare your](https://reader034.fdocuments.net/reader034/viewer/2022042119/5e9858e37a49a1288276752a/html5/thumbnails/8.jpg)
Activations
More or less all your favourite activations are available:
Sigmoid, tanh, ReLu, softplus, hard sigmoid, linear
Advanced activations implemented as a layer (after desiredneural layer)
Advanced activations: LeakyReLu, PReLu, ELU, ParametricSoftplus, Thresholded linear and Thresholded Relu
Dylan Drover STAT 946
Keras: An Introduction
![Page 9: Keras: An Introduction - University of Waterloo...Dylan Drover STAT 946 Keras: An Introduction. General Design General idea is to based on layers and their input/output Prepare your](https://reader034.fdocuments.net/reader034/viewer/2022042119/5e9858e37a49a1288276752a/html5/thumbnails/9.jpg)
Objectives and Optimizers
Objective Functions:
Error loss: rmse, mse, mae, mape, msle
Hinge loss: squared hinge, hinge
Class loss: binary crossentropy, categorical crossentropy
Optimization:
Provides SGD, Adagrad, Adadelta, Rmsprop and Adam
All optimizers can be customized via parameters
Dylan Drover STAT 946
Keras: An Introduction
![Page 10: Keras: An Introduction - University of Waterloo...Dylan Drover STAT 946 Keras: An Introduction. General Design General idea is to based on layers and their input/output Prepare your](https://reader034.fdocuments.net/reader034/viewer/2022042119/5e9858e37a49a1288276752a/html5/thumbnails/10.jpg)
Parallel Capabilities
Training time is drastically reduced thanks to Theano’s GPUsupport
Theano compiles into CUDA, NVIDIA’s GPU API
Currently will only work with NVIDIA cards but Theano isworking on OpenCL version
TensorFlow has similar support
THEANO FLAGS=mode=FAST RUN,device=gpu,floatX=float32 python your net.py
Dylan Drover STAT 946
Keras: An Introduction
![Page 11: Keras: An Introduction - University of Waterloo...Dylan Drover STAT 946 Keras: An Introduction. General Design General idea is to based on layers and their input/output Prepare your](https://reader034.fdocuments.net/reader034/viewer/2022042119/5e9858e37a49a1288276752a/html5/thumbnails/11.jpg)
Architecture/Weight Saving and Loading
Model architectures can be saved and loaded
Model parameters (weights) can be saved and loaded
Dylan Drover STAT 946
Keras: An Introduction
![Page 12: Keras: An Introduction - University of Waterloo...Dylan Drover STAT 946 Keras: An Introduction. General Design General idea is to based on layers and their input/output Prepare your](https://reader034.fdocuments.net/reader034/viewer/2022042119/5e9858e37a49a1288276752a/html5/thumbnails/12.jpg)
Callbacks
Allow for function call during training
Callbacks can be called at different points of training (batchor epoch)
Existing callbacks: Early Stopping, weight saving after epoch
Easy to build and implement, called in training function, fit()
Dylan Drover STAT 946
Keras: An Introduction
![Page 13: Keras: An Introduction - University of Waterloo...Dylan Drover STAT 946 Keras: An Introduction. General Design General idea is to based on layers and their input/output Prepare your](https://reader034.fdocuments.net/reader034/viewer/2022042119/5e9858e37a49a1288276752a/html5/thumbnails/13.jpg)
Model Type: Sequential
Sequential models are linearstack of layers
The model we all know andlove
Treat each layer as objectthat feeds into the next
Dylan Drover STAT 946
Keras: An Introduction
![Page 14: Keras: An Introduction - University of Waterloo...Dylan Drover STAT 946 Keras: An Introduction. General Design General idea is to based on layers and their input/output Prepare your](https://reader034.fdocuments.net/reader034/viewer/2022042119/5e9858e37a49a1288276752a/html5/thumbnails/14.jpg)
Dylan Drover STAT 946
Keras: An Introduction
![Page 15: Keras: An Introduction - University of Waterloo...Dylan Drover STAT 946 Keras: An Introduction. General Design General idea is to based on layers and their input/output Prepare your](https://reader034.fdocuments.net/reader034/viewer/2022042119/5e9858e37a49a1288276752a/html5/thumbnails/15.jpg)
Dylan Drover STAT 946
Keras: An Introduction
![Page 16: Keras: An Introduction - University of Waterloo...Dylan Drover STAT 946 Keras: An Introduction. General Design General idea is to based on layers and their input/output Prepare your](https://reader034.fdocuments.net/reader034/viewer/2022042119/5e9858e37a49a1288276752a/html5/thumbnails/16.jpg)
Model Type: Graph
Optimized over all outputs
Graph model allows for twoor more independentnetworks to diverge or merge
Allows for multiple separateinputs or outputs
Different merging layers(sum or concatenate)
Dylan Drover STAT 946
Keras: An Introduction
![Page 17: Keras: An Introduction - University of Waterloo...Dylan Drover STAT 946 Keras: An Introduction. General Design General idea is to based on layers and their input/output Prepare your](https://reader034.fdocuments.net/reader034/viewer/2022042119/5e9858e37a49a1288276752a/html5/thumbnails/17.jpg)
Dylan Drover STAT 946
Keras: An Introduction
![Page 18: Keras: An Introduction - University of Waterloo...Dylan Drover STAT 946 Keras: An Introduction. General Design General idea is to based on layers and their input/output Prepare your](https://reader034.fdocuments.net/reader034/viewer/2022042119/5e9858e37a49a1288276752a/html5/thumbnails/18.jpg)
Dylan Drover STAT 946
Keras: An Introduction
![Page 19: Keras: An Introduction - University of Waterloo...Dylan Drover STAT 946 Keras: An Introduction. General Design General idea is to based on layers and their input/output Prepare your](https://reader034.fdocuments.net/reader034/viewer/2022042119/5e9858e37a49a1288276752a/html5/thumbnails/19.jpg)
Example: A SUPER interesting application
Sarcasm detection in Amazon.com reviews:
Based on theory that sarcasm can be detected usingsentiment transitions
Training set was separated into sarcastic and regular reviews
Stanford recursive sentiment was run on each sentence tocreate sentiment vector
Dylan Drover STAT 946
Keras: An Introduction
![Page 20: Keras: An Introduction - University of Waterloo...Dylan Drover STAT 946 Keras: An Introduction. General Design General idea is to based on layers and their input/output Prepare your](https://reader034.fdocuments.net/reader034/viewer/2022042119/5e9858e37a49a1288276752a/html5/thumbnails/20.jpg)
In Summary
Pros:
Easy to implement
Lots of choice
Extendible and customizable
GPU
High level
Active community
keras.io
Cons:
Lack of generative models
High level
Theano overhead
Dylan Drover STAT 946
Keras: An Introduction