Implementation of a JNI Wrapper & MapReduce Compatibility Layer for StreamMine3G

by Benjamin Schmidt

Faculty of Computer Science Institute of Systems Architecture, Chair of Systems Engineering

Dresden, 25.11.13

Dresden, 25.11.13 Folie 2 von 25

Outline

• Introduction to StreamMine3G• JNI Wrapper• MapReduce Compatibility Layer• Future Tasks

Dresden, 25.11.13 Folie 3 von 25

Outline• Introduction to StreamMine3G• JNI Wrapper• MapReduce Compatibility Layer• Future Tasks

Dresden, 25.11.13 Folie 4 von 25

Introduction to StreamMine3G

Dresden, 25.11.13 Folie 5 von 25

Introduction to StreamMine3G

Dresden, 25.11.13 Folie 6 von 25

Introduction to StreamMine3G

...

...

Dresden, 25.11.13 Folie 7 von 25

Introduction to StreamMine3G

• Event processing engine• Written in C++• Provides fault tolerance at low cost• Supports stateful operators

Dresden, 25.11.13 Folie 8 von 25

Introduction to StreamMine3G

Dresden, 25.11.13 Folie 9 von 25

Introduction to StreamMine3G

• Streams – Sequence of tuples/events• Topology – Pattern of communication

between Operators

• Slices/Partitions – Can be deployed on different nodes

Dresden, 25.11.13 Folie 10 von 25

Outline• Introduction to StreamMine3G• JNI Wrapper• MapReduce Compatibility Layer• Future Tasks

Dresden, 25.11.13 Folie 11 von 25

JNI Wrapper : Introduction

Java

Dresden, 25.11.13 Folie 12 von 25

JNI Wrapper : Introduction

Dresden, 25.11.13 Folie 13 von 25

Dresden, 25.11.13 Folie 14 von 25

JNI Wrapper : Bottlenecks

• C++ more efficient execution• Additional JNI Redirections• Datatype conversion• Generation of Java Objects during

De-/Serialization C++

Dresden, 25.11.13 Folie 15 von 25

JNI Wrapper : Reasons for Java

• C++ rather difficult to implement• Most Pupils and Students learn Java from

the Beginning• Already implemented solutions• Possibly easier to debug

Dresden, 25.11.13 Folie 16 von 25

JNI Wrapper: Problems

• Datatype conversion (Pointer & String)• Thread handling• Optimization (Bytearrays)

Dresden, 25.11.13 Folie 17 von 25

Outline• Introduction to StreamMine3G• JNI Wrapper• MapReduce Compatibility Layer• Future Tasks

Dresden, 25.11.13 Folie 18 von 25

Hadoop MapReduce

• Written in Java• Cluster of Nodes• Needs Mapper & Reducer Operator• Before Reducer phase Mapper phase must →

be finished• Batch Processing

Dresden, 25.11.13 Folie 19 von 25

MapReduce Compatibility Layer

JNI Wrapper

MapReduce CL

Dresden, 25.11.13 Folie 20 von 25

Outline• Introduction to StreamMine3G• JNI Wrapper• MapReduce Compatibility Layer• Future Tasks

Dresden, 25.11.13 Folie 21 von 25

Future Tasks : Apache Pig

• Abstraction build on top of Hadoop• consists of a domain-specific dataflow

language, Pig Latin• Generates and compiles MapReduce

programs on the fly

Dresden, 25.11.13 Folie 22 von 25

Future Tasks : Apache Pig

Apache Pig Compatability Layer

Hadoop MapReduceCompatability Layer

Dresden, 25.11.13 Folie 23 von 25

Future Task : Storm

• Real-time processing• Topologies run forever• Scalable and fault-tolerant• Consists of spout, bolts, event streams and

Nimbus

Dresden, 25.11.13 Folie 24 von 25

Future Tasks : Storm

STORM OperatorCompatability Layer

STORM

PerformanceEvaluation

?

STORMOperator

STORMOperator

Dresden, 25.11.13 Folie 25 von 25

Thank you for your attention!