ODSC - Neural Networks on AWS Lambda
19
OPEN DATA SCIENCE CONFERENCE Real Time Training and Forecasting using LSTM/RNNs & AWS Lambda Anoop Vasant Kumar Data Scientist Hitachi Consulting, UK
-
Upload
anoop-vasant-kumar -
Category
Data & Analytics
-
view
377 -
download
0
Transcript of ODSC - Neural Networks on AWS Lambda
OPEN DATA SCIENCE CONFERENCE
Real Time Training and Forecasting using LSTM/RNNs & AWS Lambda
Anoop Vasant KumarData Scientist
Hitachi Consulting, UK
Understanding the problem Nuclear Power Phase Out
Switch to low carbon,
environment friendly and
affordable energy.
Forecast of Energy Price
Trading Optimization and
Energy Price Forecast
Generation Optimization
Power Generation Optimization
Project objective #1: Cost effective method to provide real time market price forecasting service.
Project objective #2: A scalable and distributed solution, that forecasts prices for each of the 100+ energy products based on live market input feeds.
High Price 48 Euro
Generation Cost 30 Euro
Low Price 10 Euro
T-5 T-4 T-3 T-3 T-2 T-1 T
Time Series of a typical energy product
Time Series of Closing Prices
Time Series of Prices of Correlated Products
Time Series of Low and High Prices
Time Series of Volume Bought/Sold Forecasted Energy Price
Machine Learning Model - Per Product
Model
A model that could learn complex nonlinear relationships
Multiple features with each input being a time series on it own
Ability to remember the past trend and sequence
Ability to consider long term dependencies in data
Ability to improve forecasts and update model parameters real time based on new data inputs
Correlation of Products
Recurrent Neural Networks are networks with loops in them, allowing information to persist.
It makes use of sequential information unlike traditional neural networks.
RNN - A network that remembers
Long Short Term Memory - Recurrent Neural NetworksRemembering long term
dependencies is their default nature.
Think of LSTMs as exactly same as RNNs except the method is which the hidden state is calculated, is slightly different!.
Time Series of Closing Prices
Time Series of Prices of Correlated Products
Time Series of Low and High Prices
Time Series of Volume Bought/Sold Forecasted Energy Price
Machine Learning Model - Per Product
LSTM - Unrolled
X
y
Time Series of Closing Prices
Time Series of Prices of Correlated Products
Time Series of Low and High Prices
Time Series of Volume Bought/Sold Forecasted Energy Price
Machine Learning Model - Per Product
Compute as-a service
Provisioning and managing of the servers to run
code.
Event-driven compute service - runs code in response
to events, such as changes to data in an Amazon
S3 bucket.
Runs code on a high-availability compute
infrastructure and performs all of the
administration.
Compute Infrastructure - AWS Lambda
High Level Architecture /Neural Networks on the Cloud
High Speed Data Connection
Scheduled Daily Lambda Trigger
Pull
Dat
a Fe
eds
and
Dat
a Pr
epro
cess
Pandas DataFrame
Prd 1Prd 2
Prd 3
Object Upload
S3 Object Upload Event Notification
Prediction Lambda Triggered Per Object Upload
Prd 120
Prd 5Prd 4
Prd 3
Prd 1Prd 2
Prd 120
Prd 1
Prediction Date2016-08-03Key ValuePrd1 36Prd2 29…Prd 120 41
Amazon API Gateway
Time Series Product 2 in Market abc
Time Series - Product a
Time Series Product b
Preprocessed Data
Predicted Prices
X
y
![[Laws] Meetup - AWS Lambda](https://static.fdocuments.net/doc/165x107/58ef2b011a28ab50758b45f1/laws-meetup-aws-lambda.jpg)
