Expanding beyond SPLIBM Streams 4.1

Multiple Language Support

1

IBM Streams Enterprise Ready

Distributed Platform

IBM Streams Application Features

SPL Java Scala Python(experimental) …

Integration with Publish & Subscribe

2

SPL Java Scala

SPL Job

Java Job

Scala JobJava Job with

SPL & Scala Elements

SPL Job

Agenda

• Streamsx.topology project

• Java Applications

• Scala Applications

• Data Interchange

• Streams Features

• SPL integration

Streamsx.topology

• A project that supports building streaming topologies (applications) for IBM Streams in different programming languages, such as Java and Scala.

https://github.com/IBMStreams/streamsx.topology

• Open Source

• Agile Development

Frequent releases

• Not tied to IBM Streams product release cycle

• 1.1 release included in IBM Streams 4.1 – November 20th

Java Applications forIBM Streams

Java Application API

• Build streaming applications for IBM Streams with Java

Natural for Java developer

Make the common case simple

Supports Java 8

Supports IBM Streams 4.0+

No SPL knowledge required

Streams of Java objects

7

TStream<String>

“#stayshere” “#vegas” “#ibminsight”

TStream<HashTag>

{“#stayshere”, 4567, 2015/10/26 03:42}

{“#ibminsight”, 8364, 2015/10/26 08:23}

{“#pbr”, 1324, 2015/10/25 16:23}

• Streams are sourced, transformed and sinked by functions implemented in Java

• Represented as:

A lambda expression (Java 8)

A method reference (Java 8)

An instance of an interface

• Anonymous class

• Class that implements the interface

Functional Programming

8

ƒ

Functional Filter

9

test( “Goal! #3lions” )= false

tuple discarded

(not present on resultant stream)

Functional Transformation (10/1)

10

apply( 2839000 “2.839E+6”)=String will appear on strings stream

Functional Transformation (10/N)

11

apply(“It is not in the stars to hold our

destiny but in ourselves”

“It”, “is ”, “not”, “in”, “the”,

“stars”, “to”, “hold”, “our”,

“destiny”, “but”, “in”,

“ourselves”

)=

Each word will appear on

words stream as a tuple

Building & DeployingJava Streaming Applications

1. Declare Topology

Building & DeployingJava Streaming Applications

1. Declare Topology

2. Submit Topology

Demonstration:Java Applications forIBM Streams

Scala Applications forIBM Streams

Scala is a JVM language

• Very good interoperability between the two languages

http://www.codecommit.com/blog/java/interop-between-java-and-scala

• Java can use Scala classes/methods

• Scala can use Java classes/methods

16

IBM Streams Scala Application support

• Scala support is therefore through the Java Application API

• API also provides implicit conversions to allow Scala functions to be used as functional elements

17

https://github.com/IBMStreams/streamsx.topology/wiki/Scala-Support

Demonstration:Scala Applications forIBM Streams

Data Interchange

• Streams instances typically contain multiple applications

Exchanging data using export/import streams

Multiple Applications

20

Ingest

App

Analytic

App

Analytic

App

Analytic

App

Decision

App

Decision

AppIngest

App

• Publish a stream by

Type – Type of the stream

Topic – A string – Recommendation to use MQTT style names

• vehicle/auto/sensor/engine/temp

• Subscribe to a stream by

Type

Topic – Single topic, no MQTT wildcard support

Publish/Subscribe

21

Interchangeable Data Types

22

SPL Java Scala

SPL Tuple

JSON

XML

String

Binary

Java Class

Scala Class

Example: Publish/Subscribe

23

SPL

Java

Scala

SPL

SPL

Java

SPL

IBM Streams Featuressupported by

Java & Scala

IBM Streams Features

• Windowing

Partitioned

• Parallel Regions

Round-robin, Key or Hash Partitioned

• Submission time values

• Fan-in, Fan-out of streams

• Checkpoint of operator state

• Resource Placement by tags

• Control over mapping of functions to processes (PEs)

• EMBEDDED – Application must only contain Java code

• DISTRIBUTED

• ANALYTICS_SERVICE – Direct submission from Java application to

• STANDALONE

• BUNDLE

• STANDALONE_BUNDLE

• TOOLKIT

• EMBEDDED_TESTER

• STANDALONE_TESTER

• DISTRIBUTED_TESTER

Execution Contexts

26

Creation of a bundle

for later submission

Testing of applications

using JUnit or similar

SPL Integration

SPL Streams within the Java Application

• Re-use of the IBM Streams Java Operator API

• Created by:

Invocations of SPL operators

Subscribing to an stream of SPL tuples

• Re-use existing SPL toolkits

C++ primitive, Java primitive, SPL composite

• Create SPL composites to wrap SPL sub-graphs

That can then be invoked from a Java/Scala application

SPL Operator Invocation

29

Summary

IBM Streamsexpanded to a wider developer pool

31

IBM Streams Enterprise Ready

Distributed Platform

IBM Streams Application Features

SPL Java Scala …

Integration across languages

• Make the simple - simple

• Re-use of existing SPL toolkits

• Variety of execution mechanisms

Python(experimental)

© 2015 IBM Corporation

Thank You