Multi-tenant Streaming and TensorFlow as a Service with Hops · Secure Structured Spark Streaming...

56
Multi-tenant Streaming and TensorFlow as a Service with Hops Big Data Conference Vilnius, Nov 2017 Hops @hopshadoop http://github.com/hopshadoop http://www.hops.io Theofilos Kakantousis Researcher @ RISE SICS Co-founder @ Logical Clocks AB Slides by Jim Dowling, Theofilos Kakantousis

Transcript of Multi-tenant Streaming and TensorFlow as a Service with Hops · Secure Structured Spark Streaming...

Page 1: Multi-tenant Streaming and TensorFlow as a Service with Hops · Secure Structured Spark Streaming App Developer 1.Discover: Schema Registry and Kafka Broker Endpoints 2.Create: Kafka

Multi-tenant Streaming and TensorFlow as a Service with Hops

Big Data Conference Vilnius, Nov 2017

Hops

@hopshadoop

http://github.com/hopshadoop

http://www.hops.io

Theofilos Kakantousis

Researcher @ RISE SICS

Co-founder @ Logical Clocks AB

Slides by Jim Dowling, Theofilos Kakantousis

Page 2: Multi-tenant Streaming and TensorFlow as a Service with Hops · Secure Structured Spark Streaming App Developer 1.Discover: Schema Registry and Kafka Broker Endpoints 2.Create: Kafka

Hadoop-as-a-Service in Sweden

• HopsWorks

• Spark/Flink/Kafka/TensorFlow/Hadoop-as-a-service

• Built on Hops Hadoop (www.hops.io)

• RISE SICS ICE

• 250 kW Datacenter, ~1000 servers

• Research and test environment

https://www.sics.se/projects/sics-ice-data-center-in-lulea2017-11-30 www.hops.io 2/48

Page 3: Multi-tenant Streaming and TensorFlow as a Service with Hops · Secure Structured Spark Streaming App Developer 1.Discover: Schema Registry and Kafka Broker Endpoints 2.Create: Kafka

Big Data and AI

2017-11-30 www.hops.io

3

Page 4: Multi-tenant Streaming and TensorFlow as a Service with Hops · Secure Structured Spark Streaming App Developer 1.Discover: Schema Registry and Kafka Broker Endpoints 2.Create: Kafka

Becoming a BigData-AI Company

2017-11-30 www.hops.io 4/48

Page 5: Multi-tenant Streaming and TensorFlow as a Service with Hops · Secure Structured Spark Streaming App Developer 1.Discover: Schema Registry and Kafka Broker Endpoints 2.Create: Kafka

Hops Hadoop

HopsYARNResource

Manager

Storage HopsFS

On-Premise AWS GCEPlatform

ProcessingElasticsearch

TensorflowSpark Flink

HopsWorks

Kibana

Logstash Kafka

Grafana

Hops meets the BigData & AI Hierarchy of Needs

Hive

2017-11-30 www.hops.io 5/48

Page 6: Multi-tenant Streaming and TensorFlow as a Service with Hops · Secure Structured Spark Streaming App Developer 1.Discover: Schema Registry and Kafka Broker Endpoints 2.Create: Kafka

HopsFS Architecture

NameNodeFile Blocks Mappings

File System Metadata

File1 Blk1 DN1, Blk2 DN4, Blk3 DN5

File5 Blk1 DN4, Blk2 DN2, Blk3 DN9

File4 Blk1 DN100

File3 Blk1 DN1, Blk2 DN2, Blk3 DN3

File2 Blk1 DN1, Blk2 DN4

FileN Blk1 DN2, Blk2 DN8

File Blocks Mappings

File1 Metadata

File2 Metadata

File Blocks Mappings

File3 Metadata

File4 Metadata

File Blocks Mappings

File5 Metadata

File6 Metadata

File Blocks Mappings

File7 Metadata

File8 Metadata

DataNodes

Distributed

Database

2017-11-30 www.hops.io 6/48

Page 7: Multi-tenant Streaming and TensorFlow as a Service with Hops · Secure Structured Spark Streaming App Developer 1.Discover: Schema Registry and Kafka Broker Endpoints 2.Create: Kafka

HopsFS: Next Generation HDFS*

16xThroughput

FasterBigger

*https://www.usenix.org/conference/fast17/technical-sessions/presentation/niazi

37xNumber of files

Scale Challenge Winner (2017)

2017-11-30 www.hops.io

7

Page 8: Multi-tenant Streaming and TensorFlow as a Service with Hops · Secure Structured Spark Streaming App Developer 1.Discover: Schema Registry and Kafka Broker Endpoints 2.Create: Kafka

www.hops.io

HopsFS now stores Small Files in the DB

8

2017-11-30

Page 9: Multi-tenant Streaming and TensorFlow as a Service with Hops · Secure Structured Spark Streaming App Developer 1.Discover: Schema Registry and Kafka Broker Endpoints 2.Create: Kafka

Hops Concepts

Hadoop•Clusters

•Users

•Jobs/Applications

•Files

•ACLs

•Sys Admins

•Kerberos

Hops•Projects

- Datasets/Topics

- Per project users

- Jobs/Notebooks

- SSL/TLS

2017-11-30 www.hops.io 9/48

Page 10: Multi-tenant Streaming and TensorFlow as a Service with Hops · Secure Structured Spark Streaming App Developer 1.Discover: Schema Registry and Kafka Broker Endpoints 2.Create: Kafka

Privacy-by-Design with Projects, Data, Users

2017-11-30 www.hops.io 10

Page 11: Multi-tenant Streaming and TensorFlow as a Service with Hops · Secure Structured Spark Streaming App Developer 1.Discover: Schema Registry and Kafka Broker Endpoints 2.Create: Kafka

Proj-42

Projects for Software-as-a-Service

A Project is a Grouping of Users and Data

Proj-X

Shared TopicTopic /Projs/My/Data

Proj-AllCompanyDB

Ismail et al, Hopsworks: Improving User Experience and Development on Hadoop with Scalable, Strongly Consistent Metadata, ICDCS 2017

2017-11-30 www.hops.io 11/48

Page 12: Multi-tenant Streaming and TensorFlow as a Service with Hops · Secure Structured Spark Streaming App Developer 1.Discover: Schema Registry and Kafka Broker Endpoints 2.Create: Kafka

IoT Project

Kafka Topic

SaaS IoT Platform: Project per Customer

ACME Project

ACME Topic

ACME HDFSDataset

Data Stream

Generic Analytics

SharedCustom Analytics

ACME manage membership

2017-11-30 www.hops.io

12

Page 13: Multi-tenant Streaming and TensorFlow as a Service with Hops · Secure Structured Spark Streaming App Developer 1.Discover: Schema Registry and Kafka Broker Endpoints 2.Create: Kafka

Project Roles

Data Owner Privileges

- Import/Export data

- Manage Membership

- Share DataSets,Topics

Data Scientist Privileges

- Write and Run code

We delegate administration of privileges to users

2017-11-30 www.hops.io

13

Page 14: Multi-tenant Streaming and TensorFlow as a Service with Hops · Secure Structured Spark Streaming App Developer 1.Discover: Schema Registry and Kafka Broker Endpoints 2.Create: Kafka

Hopsworks: Hadoop for Humans

2017-11-30 www.hops.io 14

Page 15: Multi-tenant Streaming and TensorFlow as a Service with Hops · Secure Structured Spark Streaming App Developer 1.Discover: Schema Registry and Kafka Broker Endpoints 2.Create: Kafka

Hopsworks Data Platform

Develop Train Test Deploy

Hopsworks

MySQL Cluster

Hive

InfluxDB

ElasticSearch

KafkaProjects,Datasets,Users

HopsFS / YARN

Spark, Flink, Tensorflow

Jupyter, Zeppelin

Jobs, Kibana, Grafana

RESTAPI

2017-11-30 www.hops.io

15

Page 16: Multi-tenant Streaming and TensorFlow as a Service with Hops · Secure Structured Spark Streaming App Developer 1.Discover: Schema Registry and Kafka Broker Endpoints 2.Create: Kafka

Hopsworks

www.hops.io2017-11-30 16/48

Page 17: Multi-tenant Streaming and TensorFlow as a Service with Hops · Secure Structured Spark Streaming App Developer 1.Discover: Schema Registry and Kafka Broker Endpoints 2.Create: Kafka

Manage Projects like GitHub

2017-11-30 www.hops.io 17/48

Page 18: Multi-tenant Streaming and TensorFlow as a Service with Hops · Secure Structured Spark Streaming App Developer 1.Discover: Schema Registry and Kafka Broker Endpoints 2.Create: Kafka

Share like in Dropbox

Share any Data Source/Sink: HDFS Datasets, Kafka Topics, etc

2017-11-30 www.hops.io 18/48

Page 19: Multi-tenant Streaming and TensorFlow as a Service with Hops · Secure Structured Spark Streaming App Developer 1.Discover: Schema Registry and Kafka Broker Endpoints 2.Create: Kafka

Modern Data Parallel Platforms

2017-11-30 www.hops.io 19/48

Page 20: Multi-tenant Streaming and TensorFlow as a Service with Hops · Secure Structured Spark Streaming App Developer 1.Discover: Schema Registry and Kafka Broker Endpoints 2.Create: Kafka

Workflow/Jobs and Notebook Support

2017-11-30 www.hops.io 20/48

Page 21: Multi-tenant Streaming and TensorFlow as a Service with Hops · Secure Structured Spark Streaming App Developer 1.Discover: Schema Registry and Kafka Broker Endpoints 2.Create: Kafka

Custom Python Environments with Conda

Python libraries are usable by Spark/Tensorflow

2017-11-30 www.hops.io 21/48

Page 22: Multi-tenant Streaming and TensorFlow as a Service with Hops · Secure Structured Spark Streaming App Developer 1.Discover: Schema Registry and Kafka Broker Endpoints 2.Create: Kafka

ETL Workloads (Jobs UI)

ParquetHive

Hopsworks

Jobs

trigger

Elastic

pipelines transform raw data

to structured data

HopsFS

2017-11-30 www.hops.io

22

Page 23: Multi-tenant Streaming and TensorFlow as a Service with Hops · Secure Structured Spark Streaming App Developer 1.Discover: Schema Registry and Kafka Broker Endpoints 2.Create: Kafka

HopsFS

YARN

Grafana/

InfluxDB

Elastic/

Kibana

Public Cloud or On-Premise

Parquet

Data Src

Batch Analytics

Kafka

…...MySQL

Streaming Analytics in Hopsworks

2017-11-30 www.hops.io

23

Page 24: Multi-tenant Streaming and TensorFlow as a Service with Hops · Secure Structured Spark Streaming App Developer 1.Discover: Schema Registry and Kafka Broker Endpoints 2.Create: Kafka

Look Ma, No Kerberos!

• For each project, a user is issued with a SSL/TLS(X.509) certificate for both authentication and encryption.

- Project based access on Kafka resources.

- Custom Authorizer

• Services are also issued with SSL/TLS certificates.

- Both user and service certs are signed with the same CA.

- Services extract the userID from RPCs to identify the caller.

• HADOOP-13836

• Draws on ideas from Netflix’ BLESS system

2017-11-30 www.hops.io

24

Page 25: Multi-tenant Streaming and TensorFlow as a Service with Hops · Secure Structured Spark Streaming App Developer 1.Discover: Schema Registry and Kafka Broker Endpoints 2.Create: Kafka

SSL/TLS Certificate Generation

[email protected]

Users don’t see the certificates.Users authenticate using:• LDAP• password • 2-Factor Authentication

Add/Del

Users

Distributed

Database

Insert/Remove CertsProject Mgr

RootCA

HDFSSparkKafkaYARN

Cert

Signing

RequestsIntermediate

Certificate Authority

Hopsworks

2017-11-30 www.hops.io

25

Page 26: Multi-tenant Streaming and TensorFlow as a Service with Hops · Secure Structured Spark Streaming App Developer 1.Discover: Schema Registry and Kafka Broker Endpoints 2.Create: Kafka

Distributing Certs for Spark/Flink Streaming

[email protected]

1. Launch Spark/Flink Job

Distributed

Database

YARN Private LocalResources

Spark/Flink Streaming App

2. Get certs,

service endpoints

3. YARN Job, config

6. Get Schema

7. Consume

Produce

5. Read Certs

Hopsworks

4. Materialize certs

HopsUtil

8. Authenticate

2017-11-30 www.hops.io

26

Page 27: Multi-tenant Streaming and TensorFlow as a Service with Hops · Secure Structured Spark Streaming App Developer 1.Discover: Schema Registry and Kafka Broker Endpoints 2.Create: Kafka

Making Big Data Easier to Program

HopsUtil

2017-11-30 www.hops.io 27

Page 28: Multi-tenant Streaming and TensorFlow as a Service with Hops · Secure Structured Spark Streaming App Developer 1.Discover: Schema Registry and Kafka Broker Endpoints 2.Create: Kafka

Secure Structured Spark Streaming App

Developer

1.Discover: Schema Registry and Kafka Broker Endpoints

2.Create: Kafka Properties file with certs and broker details

3.Create: Producer/Consumer using Kafka Properties

4.Download: the Schema for the Topic from the Schema Registry

5.Distribute: X.509 certs to all hosts on the cluster

6.Cleanup securelyOperations

2017-11-30 www.hops.io 28/48

Page 29: Multi-tenant Streaming and TensorFlow as a Service with Hops · Secure Structured Spark Streaming App Developer 1.Discover: Schema Registry and Kafka Broker Endpoints 2.Create: Kafka

https://github.com/hopshadoop/hops-util

Properties props = new Properties();

props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG,

brokerList);

props.put(SCHEMA_REGISTRY_URL, restApp.restConnect);

props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG,

org.apache.kafka.common.serialization.StringSerializer.class);

props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG,

io.confluent.kafka.serializers.KafkaAvroSerializer.class);

props.put("producer.type", "sync");

props.put("serializer.class","kafka.serializer.StringEncoder")

;

props.put("request.required.acks", "1");

props.put("ssl.keystore.location","/var/ssl/kafka.client.keyst

ore.jks")

props.put("ssl.keystore.password","test1234")

props.put("ssl.key.password","test1234")

ProducerConfig config = new ProducerConfig(props);

String userSchema = "{\"namespace\": \"example.avro\",

\"type\": \"record\", \"name\": \"User\"," +

"\"fields\": [{\"name\": \"name\",

\"type\": \"string\"}]}";

Schema.Parser parser = new Schema.Parser();

Schema schema = parser.parse(userSchema);

GenericRecord avroRecord = new GenericData.Record(schema);

avroRecord.put("name", "testUser");

Producer<String, String> producer = new Producer<String,

String>(config);

ProducerRecord<String, Object> message = new

ProducerRecord<>(“topicName”, avroRecord );

producer.send(data);

SparkProducer producer =

HopsUtil.getSparkProducer();

Map<String, String> message = …

sparkProducer.produce(message);

HopsUtil simplifies Secure Spark/Kafka

Lots of Hard-Coded Endpoints Here!

2017-11-30 www.hops.io 29

Page 30: Multi-tenant Streaming and TensorFlow as a Service with Hops · Secure Structured Spark Streaming App Developer 1.Discover: Schema Registry and Kafka Broker Endpoints 2.Create: Kafka

Services for Streaming Analytics

2017-11-30 www.hops.io 30/48

Page 31: Multi-tenant Streaming and TensorFlow as a Service with Hops · Secure Structured Spark Streaming App Developer 1.Discover: Schema Registry and Kafka Broker Endpoints 2.Create: Kafka

Manage & Share

• Topics

• ACLs

• Avro Schemas

Kafka Self-Service UI

2017-11-30 www.hops.io

31

Page 32: Multi-tenant Streaming and TensorFlow as a Service with Hops · Secure Structured Spark Streaming App Developer 1.Discover: Schema Registry and Kafka Broker Endpoints 2.Create: Kafka

http://mkuthan.github.io/blog/2016/09/30/spark-streaming-on-yarn/

Realtime Logs

YARN aggregates logs on job completion

No good to us for Streaming

Collect logs and make them searchable in real-

time using Logstash, Elasticsearch, and Kibana

Log4j auto-configured to write to Logstash

www.hops.io

32

Page 33: Multi-tenant Streaming and TensorFlow as a Service with Hops · Secure Structured Spark Streaming App Developer 1.Discover: Schema Registry and Kafka Broker Endpoints 2.Create: Kafka

Elasticsearch,

Logstash,

Kibana

(ELK Stack)

Realtime Logs

2017-11-30 www.hops.io 33/48

Page 34: Multi-tenant Streaming and TensorFlow as a Service with Hops · Secure Structured Spark Streaming App Developer 1.Discover: Schema Registry and Kafka Broker Endpoints 2.Create: Kafka

Graphite/

InfluxDB

and

Grafana

Resource Monitoring/Alerting

2017-11-30 www.hops.io 34/48

Page 35: Multi-tenant Streaming and TensorFlow as a Service with Hops · Secure Structured Spark Streaming App Developer 1.Discover: Schema Registry and Kafka Broker Endpoints 2.Create: Kafka

Zeppelin Notebooks

www.hops.io

35

Page 36: Multi-tenant Streaming and TensorFlow as a Service with Hops · Secure Structured Spark Streaming App Developer 1.Discover: Schema Registry and Kafka Broker Endpoints 2.Create: Kafka

Jupyter Notebooks

2017-11-30 www.hops.io 36/48

Page 37: Multi-tenant Streaming and TensorFlow as a Service with Hops · Secure Structured Spark Streaming App Developer 1.Discover: Schema Registry and Kafka Broker Endpoints 2.Create: Kafka

Per-project Conda Libs in Hopsworks

2017-11-30 www.hops.io 37/48

Page 38: Multi-tenant Streaming and TensorFlow as a Service with Hops · Secure Structured Spark Streaming App Developer 1.Discover: Schema Registry and Kafka Broker Endpoints 2.Create: Kafka

Dela* – A Global Ecosystem for Datasets

Peer-to-Peer Search and Download for Huge DataSets

(ImageNet, YouTube8M, MsCoCo, Reddit, etc)

*http://ieeexplore.ieee.org/document/7980225/ (ICDCS 2017)

www.hops.io

38

Page 39: Multi-tenant Streaming and TensorFlow as a Service with Hops · Secure Structured Spark Streaming App Developer 1.Discover: Schema Registry and Kafka Broker Endpoints 2.Create: Kafka

Becoming a BigData-AI Company

2017-11-30 www.hops.io 39/48

Page 40: Multi-tenant Streaming and TensorFlow as a Service with Hops · Secure Structured Spark Streaming App Developer 1.Discover: Schema Registry and Kafka Broker Endpoints 2.Create: Kafka

Hopsworks – TensorFlow As a Service

2017-11-30 www.hops.io 40

Page 41: Multi-tenant Streaming and TensorFlow as a Service with Hops · Secure Structured Spark Streaming App Developer 1.Discover: Schema Registry and Kafka Broker Endpoints 2.Create: Kafka

Hops Way: Unified AI and Big Data

41

IT

DataLake

GPU Cluster Compute

Kafka

Data EngineeringData Science

Project1 ProjectN

Elasticsearch

2017-11-30 www.hops.io

Page 42: Multi-tenant Streaming and TensorFlow as a Service with Hops · Secure Structured Spark Streaming App Developer 1.Discover: Schema Registry and Kafka Broker Endpoints 2.Create: Kafka

Hopsworks Machine Learning-as-a-Service

Real-Time

Feature Store

Real-Time

BatchTraining, Test,Validation, AML

ModelServing

Hive, Parquet

SparkML

Tensorflow

PythonKafka

REST API

Tensorflow

Serving

[In-Progress]

Jupyter

Experiments

Visualization

Continuous Learning

2017-11-30 www.hops.io

42

Page 43: Multi-tenant Streaming and TensorFlow as a Service with Hops · Secure Structured Spark Streaming App Developer 1.Discover: Schema Registry and Kafka Broker Endpoints 2.Create: Kafka

Deep Learning Hierarchy of Scale

DDL

AllReduce

on GPU Servers

DDL with GPU Servers

and Parameter Servers

Parallel Experiments on GPU Servers

Single GPU

Many GPUs on a Single GPU Server

Days/Hours

Days

Weeks

Minutes

Training Time for ImageNet

Hours

2017-11-30 www.hops.io

“My Model’s Training.”

Training

43

Page 44: Multi-tenant Streaming and TensorFlow as a Service with Hops · Secure Structured Spark Streaming App Developer 1.Discover: Schema Registry and Kafka Broker Endpoints 2.Create: Kafka

Lots of good GPUs > A few great GPUs

Hops

100 x Nvidia 1080Ti (DeepLearning11)

8 x Nvidia P/V100 (DGX-1)

VS

Both top (100 GPUs) and bottom (8 GPUs) cost the same: 150K Euro.

2017-11-30 www.hops.io

https://www.servethehome.com/single-root-or-dual-root-for-deep-learning-gpu-to-gpu-systems

https://www.microway.com/hpc-tech-tips/comparing-nvlink-vs-pci-e-nvidia-tesla-p100-gpus-openpower-servers/

Page 45: Multi-tenant Streaming and TensorFlow as a Service with Hops · Secure Structured Spark Streaming App Developer 1.Discover: Schema Registry and Kafka Broker Endpoints 2.Create: Kafka

HopsFS

YARN

Public Cloud or On-Premise

Parquet

Business Intelligence Workloads

Hive

Jupyter/Zeppelin

or Jobs

Kibana

reports

Zeppelin

2017-11-30 www.hops.io

46

Page 46: Multi-tenant Streaming and TensorFlow as a Service with Hops · Secure Structured Spark Streaming App Developer 1.Discover: Schema Registry and Kafka Broker Endpoints 2.Create: Kafka

HopsFS

YARN

FeatureStoreTensorflow

Serving

Public Cloud or On-Premise

Tensorboard

Deep Learning in Hopsworks

Experiments

Kafka

Hive

2017-11-30 www.hops.io

47

Page 47: Multi-tenant Streaming and TensorFlow as a Service with Hops · Secure Structured Spark Streaming App Developer 1.Discover: Schema Registry and Kafka Broker Endpoints 2.Create: Kafka

Spark & TensorFlow – Cluster Integration

Training Data and Model Store

Cluster Manager

Single GPU

Experiment

Parallel Experiments

(HyperParam Tuning)

Distributed

Training JobDeprecated

Mix of commodity GPUs and more powerful

GPUs good for (1) parallel experiments and

(2) distributed training

TFoSpark

Horovod

2017-11-30 www.hops.io

Hops

48

Page 48: Multi-tenant Streaming and TensorFlow as a Service with Hops · Secure Structured Spark Streaming App Developer 1.Discover: Schema Registry and Kafka Broker Endpoints 2.Create: Kafka

GPU Resource Requests in Hops

HopsYARN (Supports GPUs-as-a-Resource)

4 GPUs on any host

10 GPUs on 1 host

100 GPUs on 10 hosts with ‘Infiniband’

20 GPUs on 2 hosts with ‘Infiniband_P100’

HopsHopsFS

2017-11-30 www.hops.io

49

Page 49: Multi-tenant Streaming and TensorFlow as a Service with Hops · Secure Structured Spark Streaming App Developer 1.Discover: Schema Registry and Kafka Broker Endpoints 2.Create: Kafka

GPUs supported as a Resource in Hops 2.8.2*

Hops is the only Hadoop distribution to support GPUs-as-a-Resource.

*Robin Andersson, GPU Integration for Deep Learning on YARN, MSc Thesis, 2017

2017-11-30 www.hops.io

50

Page 50: Multi-tenant Streaming and TensorFlow as a Service with Hops · Secure Structured Spark Streaming App Developer 1.Discover: Schema Registry and Kafka Broker Endpoints 2.Create: Kafka

HopsUtil

• pip install hops

•Distributed TensorFlow in Python

- Parameter sweeps for ML experiments

- Managing tensorboard, Load/save models in HopsFS

from hops import tflauncher

args_dict = {'steps': [500, 2500, 5000, 7500, 5000]}

tflauncher.launch(spark, training, args_dict)

2017-11-30 www.hops.io 51/55

Page 51: Multi-tenant Streaming and TensorFlow as a Service with Hops · Secure Structured Spark Streaming App Developer 1.Discover: Schema Registry and Kafka Broker Endpoints 2.Create: Kafka

TensorFlow/Hops on 10 1080Ti GPUs

2017-11-30 www.hops.io

52

Page 52: Multi-tenant Streaming and TensorFlow as a Service with Hops · Secure Structured Spark Streaming App Developer 1.Discover: Schema Registry and Kafka Broker Endpoints 2.Create: Kafka

Tensorboard Integration

2017-11-30 www.hops.io

53

Page 53: Multi-tenant Streaming and TensorFlow as a Service with Hops · Secure Structured Spark Streaming App Developer 1.Discover: Schema Registry and Kafka Broker Endpoints 2.Create: Kafka

Demo!

Hops

2017-11-30 www.hops.io

Page 54: Multi-tenant Streaming and TensorFlow as a Service with Hops · Secure Structured Spark Streaming App Developer 1.Discover: Schema Registry and Kafka Broker Endpoints 2.Create: Kafka

Summary

•Europe’s Only Hadoop Distribution – Hops Hadoop

- Fully Open-Source

- Supports larger/faster Hadoop Clusters

•Hopsworks is a new Data Platform built on HopsFS with first-class support for Streaming and ML

- Spark or Flink

- TensorFlow

- Support services for Streaming Analytics and ML-Deep Learning

2017-11-30 www.hops.io

Page 55: Multi-tenant Streaming and TensorFlow as a Service with Hops · Secure Structured Spark Streaming App Developer 1.Discover: Schema Registry and Kafka Broker Endpoints 2.Create: Kafka

The Team

Jim Dowling, Seif Haridi, Tor Björn Minde, Gautier Berthou, Salman Niazi, Mahmoud Ismail, Theofilos Kakantousis, Ermias Gebremeskel, Antonios Kouzoupis, Alex Ormenisan, Fabio Buso, Robin Andersson, August Bonds, Filotas Siskos, Mahmoud Hamed.

Active:

Alumni:

Vasileios Giannokostas, Johan Svedlund Nordström,Rizvi Hasan, Paul Mälzer, Bram Leenders, Juan Roca, Misganu Dessalegn, K “Sri” Srijeyanthan, Jude D’Souza, Alberto Lorente, Andre Moré, Ali Gholami, Davis Jaunzems, Stig Viaene, Hooman Peiro, Evangelos Savvidis, Steffen Grohsschmiedt, Qi Qi, Gayana Chandrasekara, Nikolaos Stanogias, Daniel Bali, Ioannis Kerkinos, Peter Buechler, Pushparaj Motamari, Hamid Afzali, Wasif Malik, Lalith Suresh, Mariano Valles, Ying Lieu, Fanti Machmount Al Samisti, Braulio Grana, Adam Alpire, Zahin Azher Rashid, ArunaKumari Yedurupaka, Tobias Johansson , Roberto Bampi.

www.hops.io@hopshadoop

Page 56: Multi-tenant Streaming and TensorFlow as a Service with Hops · Secure Structured Spark Streaming App Developer 1.Discover: Schema Registry and Kafka Broker Endpoints 2.Create: Kafka

Thank You.

Follow us: @hopshadoop

Star us: http://github.com/hopshadoop/hopsworks

Join us: http://www.hops.io

Thank You.

Follow us: @hopshadoop

Star us: http://github.com/hopshadoop/hopsworks

Join us: http://www.hops.io

Hops