DEEP

LEARNING

ON HADOOP

ADAM GIBSON // DL4J

presentation@

Scale out Deep Learning on YARN

Next.ML

deeplearning4j.org

presentation@

ADAM / DL4J

● Adam:

○ DL specialist, in residence at Galvanize

○ Head programmer in DL4J project

● SkyMind

○ Commercial support arm of Deeplearning4j

● Deep Learning for Java (DL4J)

○ a distributed, open-source deep-learning framework

for the JVM

○ has most major deep neural nets

deeplearning4j.org

presentation@

OVERVIEW

● What is Deep Learning?

● DL major types of Neural Networks:

a. Feed-forward NN

b. Recurrent NN

c. Convolutional NN

d. Recursive NN

● Scaling DL4J with HADOOP

● Demo

deeplearning4j.org

What is DL?Machine Perception, Pattern Recognition

deeplearning4j.org

presentation@

WHY IS DL HARD?

We see this Machines see this… (Where’s the cat?)

deeplearning4j.org

presentation@

WHAT’S IT GOOD FOR?

● Recommendation engines: anticipate what you will buy or click

● Anomaly detection:

○ fraud in e-commerce

○ tumors in X-rays

○ loans’ likely to default

● Signal processing:

○ customer lifetime value estimation

○ necessary inventory

○ approaching market crash

● Image

deeplearning4j.org

presentation@

Google, Nov. 17 2014

“TWO PIZZAS SITTING ON A STOVETOP”

deeplearning4j.org

presentation@

FACIAL RECOGNITION &

FEATURE HIERARCHY

DEEP means “high level

abstractions achieved

by using model

architectures composed

of multiple non-linear

transformations”, typically

traditional Machine

Learning algorithms,

mostly Neural-Network

based

(source: Wikipedia)

deeplearning4j.org

presentation@

DL4J FACIAL RECONSTRUCTIONS

deeplearning4j.org

presentation@

LEARNING PROPERTIES OF DL

● They can learn unsupervised data

● They save data scientists months of work

● They eat vectors and learn patterns based on occurrence

statistics

● They can handle millions of parameters at a time

● After training, they can be deployed for fast classification

● High level representation are small vectors, achieving

deeplearning4j.org

MAJOR TYPES OF

DL NETWORKS

deeplearning4j.org

presentation@

FEED FORWARD NN

● Perceptron (1957)

● Back-prop learning

● No loops

● Can have different

topologies, typically

multi layered

deeplearning4j.org

presentation@

RECURRENT NN

● Loops

● Temporal

behavior

● Used for

temporal

series

deeplearning4j.org

presentation@

CONVOLUTIONAL NN

Specific type of Feed-Forward NN, impressive recent results

● Neocognitron (Fukushima’80)

● Improvement (LeCun’98)

● Google Brain (Ng’11)

deeplearning4j.org

presentation@

RECURSIVE NN

● Recursive AutoencoderDeals with variable-length input, has a topological structure, good for NLP

deeplearning4j.org

presentation@

RECURSIVE NN

● Recursive Neural Tensor Network

deeplearning4j.org

SCALING WITH

HADOOP AND

YARN

deeplearning4j.org

presentation@

SGD: Serial vs. Parallel

Model

Training Data

Worker 1

Master

Partial

Model

Global Model

Worker 2

Partial Model

Worker N

Partial

Model

Split 1 Split 2 Split 3

…

deeplearning4j.org

presentation@

WHAT’S ?

● Everything is in the cloud now, right?

deeplearning4j.org

presentation@

MapReduce vs. Parallel iterative

Input

Output

Processor Processor Processor

Superstep 1

Processor Processor

Superstep 2

. . .

Processor

deeplearning4j.org

DEMO TIME

deeplearning4j.org

Thank you