Convolutional Neural Networks at scale in Spark MLlib

download Convolutional Neural Networks at scale in Spark MLlib

of 23

  • date post

    28-Jan-2018
  • Category

    Technology

  • view

    963
  • download

    1

Embed Size (px)

Transcript of Convolutional Neural Networks at scale in Spark MLlib

  1. 1. Spark Technology Center Convolutional Neural Networks at Scale in MLlib Jeremy Nixon
  2. 2. Spark Technology Center 1. Machine Learning Engineer at the Spark Technology Center 2. Contributor to MLlib, dedicated to scalable deep learning. 3. Previously, studied Applied Mathematics to Computer Science and Economics at Harvard Jeremy Nixon
  3. 3. Future Work 1. Convolutional Neural Networks a. Convolutional Layer Type b. Max Pooling Layer Type 2. Flexible Deep Learning API 3. More Modern Optimizers a. Adam b. Adadelta + Nesterov Momentum 4. More Modern activations 5. Dropout / L2 Regularization 6. Batch Normalization 7. Tensor Support 8. Recurrent Neural Networks (LSTM)
  4. 4. Spark Technology Center 1. Framing Deep Learning 2. MLlib Deep Learning API 3. Optimization 4. Performance 5. Future Work Structure
  5. 5. Spark Technology Center 1. Structural Assumptions 2. Automated Feature Engineering 3. Learning Representations 4. Applications Framing Convolutional Neural Networks
  6. 6. Spark Technology Center - Network depth creates an extraordinary range of possible models. - That flexibility creates value in large datasets to reduce variance. Structural Assumptions: Combinatorial Flexibility
  7. 7. Spark Technology Center X = Normalized Data, W1 , W2 = Weights Forward: 1. Multiply data by first layer weights | (X*W1 ) 2. Put output through non-linear activation | max(0, X*W1 ) 3. Multiply output by second layer weights | max(0, X*W1 ) * W2 4. Return predicted output Structural Assumption: The Model
  8. 8. Spark Technology Center - Pixels - Edges - Shapes - Parts - Objects - Learn features that are optimized for the data - Makes transfer learning feasible Structural Assumptions: Hierarchical Abstraction
  9. 9. Spark Technology Center Structural Assumptions: Location Invariance - Convolution is a restriction on the features that can be combined. - Location Invariance leads to strong accuracy in vision, audio, and language. colah.github.io
  10. 10. Spark Technology Center Automated Feature Engineering
  11. 11. Spark Technology Center Learning Representations Hidden Layer + Nonlinearity http://colah.github.io/posts/2014-03-NN-Manifolds-To pology/
  12. 12. Spark Technology Center 1. CNNs - State of the art a. Object Recognition b. Object Localization c. Image Segmentation d. Image Restoration e. Music Recommendation 2. RNNs (LSTM) - State of the Art a. Speech Recognition b. Question Answering c. Machine Translation d. Text Summarization e. Named Entity Recognition f. Natural Language Generation g. Word Sense Disambiguation h. Image / Video Captioning i. Sentiment Analysis Applications
  13. 13. Spark Technology Center Flexibility. High level enough to be efficient. Low level enough to be expressive. MLlib Flexible Deep Learning API
  14. 14. Spark Technology Center Flexibility. High level enough to be efficient. Low level enough to be expressive. MLlib Flexible Deep Learning API
  15. 15. Spark Technology Center Modularity enables Logistic Regression, Feedforward Networks. MLlib Flexible Deep Learning API
  16. 16. Spark Technology Center Introducing Convolutional and Max-Pooling Layer types. MLlib Convolutional Neural Network
  17. 17. Spark Technology Center Optimization
  18. 18. Spark Technology Center Optimization
  19. 19. Spark Technology Center Parallel implementation of backpropagation: 1. Each worker gets weights from master node. 2. Each worker computes a gradient on its data. 3. Each worker sends gradient to master. 4. Master averages the gradients and updates the weights. Distributed Optimization
  20. 20. Spark Technology Center Parallel MLP on Spark with 7 nodes ~= Caffe w/GPU (single node). Advantages to parallelism diminish with additional nodes due to communication costs. Additional workers are valuable up to ~20 workers. See https://github.com/avulanov/ann-benc hmark for more details Performance
  21. 21. Spark Technology Center Github: https://github.com/JeremyNixon/sparkdl Spark Package: https://spark-packages.org/package/JeremyNixon/s parkdl Access
  22. 22. Spark Technology Center 1. GPU Acceleration (External) 2. Keras Integration 3. Residual Layers 4. Hardening 5. Regularization 6. Batch Normalization 7. Tensor Support Future Work
  23. 23. Spark Technology Center Thank you for your attention! Questions?