Webinar: Extending WSO2 Analytics Platform

Mohanadarshan VivekanandalingamAssociate Technical Lead

Agenda

● Introduction to WSO2 analytics platform

● Examine extensions including

○ Real-time analytics (Siddhi extension)○ Batch analytics extensions○ Event Receiver and Event Publisher extensions○ Predictive analytics extensions

● Outline the benefits of WSO2’s analytics platform through

real-world customer use cases.

2

WSO2 Data Analytics Server

3

WSO2 Data Analytics Server

4

• Fully-open source solution with the ability to build systems and applications that collect and analyze both realtime and persisted data and communicate the results.

• High performance data capture framework

• Highly available and scalable by design

• Pre-built Data Agents for WSO2 products

Real-time Analytics

5

Realtime Analytics Extensions

6

●This includes Siddhi Extensions

■ Custom Function

■ Custom Window

■ Custom Aggregate

■ Custom Stream Function

■ Custom Stream Processor

Function Extension

7

● Consumes zero or more parameters for each event and output a single

attribute as an output.

● This could be used to manipulate event attributes to generate new

attribute like Function operator.

● Extend org.wso2.siddhi.core.executor.function.FunctionExecutor

from InValueStream

select math:sin(inValue) as sinValue

insert into OutMediationStream;

Window Extension

8

● Allows events to be collected and expired without altering the event

format based on the given input parameters like the Window operator.

● Default Window types - Length, Time, Unique and etc..

● Extend org.wso2.siddhi.core.query.processor.stream.window.WindowProcessor

from TempStream#window.custom:customWindow(10)

select *

insert into AvgRoomTempStream ;

Aggregate Extension

9

● Consumes zero or more parameters for each event and output a single

attribute (having an aggregated results based in the input parameters as

an output).

● Used with conjunction with a window in order to find the aggregated

results based on the given window.

● Default Aggregators - sum, max, avg and etc..

● Extend org.wso2.siddhi.core.query.selector.attribute.aggregator.AttributeAggregator

from pizzaOrder#window.length(20)

select custom:count(orderNo) as totalOrders

insert into orderCount;

Stream Function Extension

10

● Allows events to be altered by adding one or more attributes to it. (Simply,

can output multiple outputs)

● Events can be output upon each event arrival

● Extend org.wso2.siddhi.core.query.processor.stream.function.StreamFunctionProcessor

from geocodeStream#geo:geocode(location)

select latitude, longitude, formattedAddress

insert into dataOut;

Stream Processor Extension

11

● Allows to alter an event format

● Considered as Window++

● Extend org.wso2.siddhi.core.query.processor.stream.StreamProcessor

from baseballData#timeseries:regress(2, 10000, 0.95, salary,

rbi, walks, strikeouts, errors)

select *

insert into regResults;

Batch Analytics Extension

12

•User Defined Functions (UDF)

•Aggregators for Lucene Indexing

•DataSource Connectors (Eg: HBase, Cassandra & etc..)

User Defined Functions (UDF)

13

● Apache Spark allows UDFs (User Defined Functions) to be created if you

want want to use a feature that is not available for Spark by default.

public class StringConcatonator implements CarbonUDF {

/**

This UDF returns the concatenation of two strings

*/

public String concat(String firstString, String secondString) {

return firstString + secondString;

}

}

• Add below to DAS_HOME/repository/conf/analytics/spark/spark-udf-config.xml

<udf-configuration>

<custom-udf-classes> <class-name>org.wso2.customUDFs.

StringConcatonator</class-name>

...

</custom-udf-classes>

</udf-configuration>

Aggregators for Lucene Indexing

14

WSO2 DAS contains 5 default Lucene based aggregated functions.

● MIN● MAX● SUM● AVG● COUNT

Users can add custom aggregator function for Lucene by extending below interface.

org.wso2.carbon.analytics.dataservice.core.indexing.aggregates.AggregateFunction

(DAS 3.1.0 onwards)

Refer mail thread - [Architecture] [Analytics] Improvements to Lucene based Aggregate functions (Installing Aggregates as OSGI components)

Datasource Connectors

15

DAS supports below datasource connectors by default.

● RDBMS● Cassandra● HBASE● HDFS

Extension can be written by implementing the below interface,

org.wso2.carbon.analytics.datasource.core.rs.AnalyticsRecordStore

https://docs.wso2.com/display/DAS310/Configuring+Data+Persistence

Predictive Analytics

16

Predictive Analytics Extensions

17

•Dataset Processors

•Input Adapters

•Model Builders

•Output Adapters

Input Adapters

18

● Used to read data from different storages such as files, HDFs and registry.

● Can create an ML Input Adapter by implementing the MLInputAdapter

interface.

Dataset Processors

19

● Each data source should have an implementation of DatasetProcessor.

● ML supports File, HDFS and DAS as data sources. Therefore we have the

following implementation classes.

Model Builders

20

● ML model generation can be extended by implementing

MLModelBuilders.

● Currently we have a supervised spark model builder and an unsupervised

spark model builder.

● If you need to extend model generation to some other library or a new

algorithm type, you can use this extension point of WSO2 ML.

Output Adapters

21

● Used to write data to different storages such as files, HDFS and registry.

● Can create an ML Output Adapter by implementing the MLOutputAdapter

interface.

Event Receiver Extensions

22

● Allows to receive events from different data sources..

● Implemented with OSGI whiteboard pattern.

Event Publisher Extensions

23

•Allows to push events to various data sinks.

•Implemented with OSGI whiteboard pattern.

Case Studies from Real Customers

24

Pacific ControlsPacific Controls is an innovative company delivering an IoT platform of platforms: Galaxy 2021. The platform allows to manage all kinds of devices within a building and take automated decisions such as moving an elevator or starting the air conditioning based on certain conditions. Within Galaxy2021, CEP is used for monitoring alarms and specific conditions.Pacific Controls also uses other products from the WSO2 platform, such as WSO2 ESB and Identity Server.

https://www.youtube.com/watch?v=OG0N7cfaJ_8

UBER

http://www.infoq.com/presentations/uber-stream-processing

UBER uses WSO2 CEP to detect fraud. P.S : Does not pay for us (Opensource at work ! ).

27

A leading Airlines uses CEP to enhance customer experience by calculating the average time to reach their boarding gate (going through security, walking, etc.). They also want to track the time it takes to clean a plane, in order to better streamline the boarding process and notify both the airline and customers about potential delays. They evaluated WSO2 CEP first as they were already using our platform and decided to use it as it addressed all their requirements.

The Cleveland Clinic, ranked among the top 3 hospitals in the US, uses a Clinical Intelligence Platform that combines big data storage, stream and batch processing to provide decision support to clinicians. Real-time analytics for the platform is provided by WSO2 CEP along with custom extensions to handle healthcare data.

Few more use cases

28

29

US Election Monitor https://wso2.com/election2016/

SUPER BOWL 50 - BigData Game

http://wso2.com/landing/big-data-game/

31

Fraud Detection

31

• Use or change the generic rules we provide and add as many rules as they like

• Change weights of Fraud Scoring Model to suit their business needs

• Use the Markov Modelling and Clustering capabilities to learn unknown Fraud Patterns in their domain

• Use the dashboard provided or plug the Fraud Detection Toolkit to their own Fraud Detection UI

http://wso2.com/library/webinars/2015/02/catch-them-in-the-act-fraud-detection-with-wso2-cep-and-wso2-bam/

https://www.youtube.com/watch?v=aLwG4thHOXg

ESB Analytics

ESB Analytics can be used to collect statistics, debug, and profile your mediation sequences.

https://docs.wso2.com/display/ESB500/ESB+Analytics

Conclusion

● Next WSO2 Analytics Platform release contains many bug fixes, improvements and features.○ Incremental Processing - Batch Analytics○ Siddhi Performance Improvements - Realtime Analytics○ Siddhi Debugger ○ Analytics features for ESB, APIM, IS, IOT and etc..○ Cross Tenant Data Retrieval in Super Tenant Spark Queries○ Custom Lucene Aggregators

● Stay tuned for next release and related updates.

33

CONTACT US !