Multi-tenant Deep Learning and Streaming as-a-Service with ... · Lifecycle of a Streaming Job...

Multi-tenant Deep Learning and Streaming as-a-Service with HopsworksTheoflos Kakantousis (@theofloskak)COO – Logical Clocks AB

Big Data Moscow 2018

©2018 Logical Clocks AB. All Rights Reserved2

Deep Learning & Streaming-as-a-Service in Sweden

● Hopsworks

– Spark/Flink/Kafka/TensorFlow/Hadoop-as-a-service

– Built on Hops Hadoop (www.hops.io)

– hops.site, 600+ users as of September 2018 ● RISE SICS ICE

– 250 kW Datacenter, ~1000 servers

https://www.sics.se/projects/sics-ice-data-center-in-lulea

http://www.hops.io/

https://www.hops.site/

https://www.sics.se/projects/sics-ice-data-center-in-lulea


[…] the general consensus seems to be that everyoneexpects some gain in performance numbers if the dataset size is increased dramatically [...]

Deep Learning needs Big data

Sun et Al. - Revisiting Unreasonable Efectiveness of Data in Deep Learning Era - 2017

Joel et Al. - Deep Learning Scaling is Predictable, Empirically - 2017


AI Hierarchy of Needs

DataEngineers

DataScientists

DataScientists?

DDL(Distributed

Deep Learning)

Deep Learning, RL

Machine Learning (ML)

Data Analytics

Data Pipelines

Big Data

Lots of GPUs

GPUs

Full-stack Data Science


Hopsworks

Hopsworks

Rest API


Hopsworks

Develop Train Test Deploy

MySQL Cluster

Hive

InfuxDB

ElasticSearch

KafkaProjects,Datasets,Users

HopsFS / YARN

Spark, Flink, Tensorfow

Jupyter

Jobs, Kibana, Grafana

RESTAPI

Big data needs scalable storage


HopsFS*

Metadata

Datanode

Namenode

● HDFS derivative with distributed metadata

– 37x increased capacity– 16x increased

throughput

HDFS Client

HDFS Client

Scale-out all layers

* HopsFS - https://goo.gl/yFCsGc

Scale Challenge Winner (2017)

Hops

https://goo.gl/yFCsGc

https://goo.gl/yFCsGc


HopsFS support for Small Files *

RAMNVMe Disk

Datanode

Namenode

> 64KB (Configurable)

< 1KB 1KB < > 64KB

● Integrates NVMe ● Open Images Dataset:

– 9m images– ~80% small fles (<64 KB)

NVMe Disk

Metadata layer - NDB

*Size Matters: Improving the Performance of Small Files in Hadoop, Middleware 2018. Niazi et al

Multi-tenancy


Projects for Software-as-a-Service

Proj-42 Proj-X

Shared TopicTopic /Projs/My/Data

Proj-AllCompanyDB


Manage Projects like Github


Share like in Dropbox

Share any Data Source/Sink: HDFS Datasets, Kafka Topics, etc


Project Authorization

● Data Owner Privileges– Import/Export data– Manage Membership– Share DataSets, Topics

● Data Scientist Privileges

– Write and Run code● Delegate Administration of

privileges to users


Custom Python environments with Conda

Python libraries are usable by Spark/Tensorfow


TLS (not Kerberos) for security in Hops

● X.509 Certifcates for authentication● 1 Certifcate for each project

user● New App certifcate

generated for each job

● Store an audit trail of the operations (read/write/create/etc) users and apps perform on HopsFs

Resource Manager

Node Manager

HopsFs

Generate App Cert

Auth w/ App Cert

Project_user cert


TLS certifcate generation

[email protected]

Users don’t see the certifcates,authenticate using:• LDAP• password • 2-Factor Authentication

Add/DelUsers

Distributed Database

Insert/Remove CertsProject Mgr

RootCA

HDFSSparkKafkaYARN

Cert Signing Requests

IntermediateCertifcate Authority

Hopsworks

Streaming-as-a-Service


ETL Workloads

ParquetHive

Hopsworks Jobs

trigger

Elastic

pipelines transform raw datato structured data

HopsFS


Streaming Analytics in Hopsworks

HopsFS YARN

HopsFS YARN

Grafana/InfluxDBGrafana/InfluxDB

Elastic/KibanaElastic/Kibana

Public Cloud or On-PremisePublic Cloud or On-Premise

Parquet / ORC

Data Src

Batch Analytics

Kafka

…...MySQLMySQL


Lifecycle of a Streaming Job

Developer

1.Discover: Schema Registry and Kafka Broker Endpoints2.Create: Kafka Properties file with certs and broker

details3.Create: Producer/Consumer using Kafka Properties

4.Download: the Schema for the Topic from the Schema Registry

5.Distribute: X.509 certs to all hosts on the cluster6.Cleanup securely

Operations

Facilitate dev+ops with hops-util https://github.com/logicalclocks/hops-util

https://github.com/logicalclocks/hops-util


Kafka Self-Service UI

Manage & Share• Topics• ACLs• Avro Schemas

Manage & Share• Topics• ACLs• Avro Schemas


Realtime Logs

● YARN aggregates logs on job completion– No good to us for Streaming

● Collect logs and make them searchable in real-time using Filebeat, Logstash, Elasticsearch, and Kibana


Realtime Logs


Resource Monitoring/Alerting


Jupyter Notebooks


Dela* – A Global Ecosystem for Datasets

Peer-to-Peer Search and Download for Huge DataSets(ImageNet, YouTube8M, MsCoCo, Reddit, etc)

*http://ieeexplore.ieee.org/document/7980225/ (ICDCS 2017)

http://ieeexplore.ieee.org/document/7980225/

ML & Deep Learning-as-a-Service


HopsML Pipeline


HopsML Spark/TensorFlow Arch

Executor/Tf Executor/Tf

Driver

HopsFSTensorBoard Model Serving

Conda Envs

Conda Envs

Distributed Training


Deep Learning Hierarchy of Scale

DDLAllReduce

on GPU Servers

DDL with GPU Serversand Parameter Servers

Parallel Experiments on GPU Servers

Single GPU

Many GPUs on a Single GPU Server

Days/Hours

Days

Weeks

Minutes

Training Time for ImageNet

Hours

“My Model’s Training.”

Training


GPU Resource Requests in HopsYARN

HopsYARN HopsYARN

10 GPUs on 1 host

100 GPUs on 10 hosts with ‘Infiniband’

Hops supports a Hetrogenous Mix of GPUs

4 GPUs on any host

Experiments in Hopsworks


The boring part of the job

● Find good Hyperparameters for your model

● Test diferent confgurations● Automate this!

“I have to run a hundred experiments to fnd the best

model,” he complained, as he showed me his Jupyter notebooks.

“That takes time. Every experiment takes a lot of

programming, because there are so many diferent parameters.

[https://thomaswdinsmore.com/2018/01/30/predictions-for-2018/ ]


Experiments in TensorFlow/Hopsworks

● Run and evaluate multiple models in parallel on a subset of the dataset

Experiment 1 Experiment 2

Experiment 4Experiment 3


Reproducible Experiments

● Results tracking● Hyperparameter tracking● Jupyter notebook versioning● Conda Env versioning● WIP: Dataset versioning


Experiments Dashboard


TensorBoard (1)


TensorBoard (2)


HopsAPI*

● Python (also Java/Scala)– Manage TensorBoard, load/save models in HDFS – TensorFlow, Horovod, TensorFlowOnSpark– Parallel experiments

● Gridsearch● Model Architecture Search with Genetic Algorithms

– Secure Streaming Analytics with Kafka/Spark/Flink– SSL/TLS certs, Avro Schema, Endpoints for

Kafka/Zookeeper/etc.

* https://github.com/logicalclocks/hops-util-py

https://github.com/logicalclocks/hops-util-py

Model Serving


Standard serving infrastructure

Scale model serving with Kubernetes

Considered best practice by the community

Provide tools to easily:● Fault tolerance● Rolling release new models● Autoscaling


Model Monitoring

HopsFS

Serving infrastructure

Re-train and deploy new model

Model monitoring infrastructure

● Log model inference requests/results to Kafka● Spark monitors model performance and input data● When to retrain?


Model Serving on Kubernetes


Orchestrating Hops workfows

Data Collection

Experimentation Training ServingFeature

Extraction

Data Transformation & Verifcation

Test

Airflow (Hopsworks Operator)


Summary

● Build a single platform to cover the entire AI hierarchy of needs.

● Increase productivity of Data Scientists – Manage all your data pipelines and workflows

under a single roof– Have first-class support for Python / Streaming/

Deep Learning / ML / Data Governance / GPUs

Hopsworks → logicalclocks.comGitHub → github.com/logicalclocksTwitter → @logicalclocks

Multi-tenant Deep Learning and Streaming as-a-Service with ... · Lifecycle of a Streaming Job...

Documents

Transcript of Multi-tenant Deep Learning and Streaming as-a-Service with ... · Lifecycle of a Streaming Job...