Harnessing Big Data with KNIME · Big Data, IoT, and the three V Variety: –KNIME inherently...

Harnessing Big Data with KNIME

Tobias Kötter

KNIME.com

Agenda

• The three V’s of Big Data

• Big Data Extension and Databases Nodes

• Demo

Variety, Volume, Velocity

Variety:• integrating heterogeneous data (and tools)

Volume:• from small files...

• ...to distributed data repositories (Hadoop)

• bring the tools to the data

Velocity:• from distributing computationally heavy

computations...

• ...to real time scoring of millions of records/sec.

Variety

• Data Integration– Small (Ascii)

– Proprietary (XLS, SAS...)

– Medium (Databases)

– Large (Hive, Impala, ParStream, HP Vertica...)

– Diverse (Numbers, Texts, Images, Networks, Sequences...)

• Tool Integration– Native

– Legacy, Inhouse

– R, Python, Matlab, ...

Volume

Every Minute…

Big Data Support

• KNIME Database Nodes

– in database processing

– preconfigured connectors

• KNIME Big Data Extension

– package required drivers/libraries for specific HDFS, Hive, Impala access

• Spark MLlib integration (coming soon)

Velocity

• Computationally Heavy Analytics:

– Distributed Execution of one workflow branch

– Parallel Execution of workflow branches

• Hosted Analytics/Prediction

– Web service Deployment of Workflows

• High Demand Scoring/Prediction:

– Continuous Execution of Workflow parts

– High Performance Scoring using generic Workflows

– High Performance Scoring of Predictive Models

KNIME Cluster Execution: Distributed Data

KNIME Cluster Execution: Distributed Analytics

Deployed Workflows

Application Access• Custom API• WSDL/SOAP based

Continuous Scoring using Workflows

• Exposes workflow fragment as RESTful web service

• Deployed on KNIME Server (v4.0 – 1H2015)

High Performance Scoring via Workflows

• Streaming Executor

• Deployed via KNIME Server (v4.1 – 2H2015/2016)

• Record (or small batch) based processing

• Exposed as RESTful web service

High Performance Scoring using Models

• Deployed on KNIME Server (v4.0 – 1H2015)

• KNIME PMML Scoring via compiled PMML

• Exposed as RESTful web service

• Partnership with Zementis

– ADAPA Real Time Scoring

– UPPI Big Data Scoring Engine

Big Data, IoT, and the three V

Variety:– KNIME inherently well-suited: open platform

– broad data source/type support

– extensive tool integration

Volume:– Big Data Extensions cover Hadoop based data integration and

aggregation

– Big Data Executors will address model building and streaming execution

Velocity:– Distributed Execution of heavy workflows to...

– High Performance Scoring of predictive models.

Big Data Extension and Database Nodes

Database Port Types

Database JDBC Connection Port (light red)• Connection information

Database Connection Port (dark red)• Connection information• SQL statement

Database Connection Ports can be connected to

Database JDBC Connection Ports but not vice versa

Database JDBC Connection Port View

Database Connection Port View

Copy SQL statement

Database Connectors

• Nodes to connect to specific Databases – Bundling necessary JDBC drivers

– Easy to use

– DB specific behavior/capability

• Hive and Impala connector part of the commercial Big Data Extension

• General Database Connector– Can connect to any JDBC source

– Register new JDBC driver viapreferences page

Register JDBC Driver

Open KNIME and go toFile -> Preferences

Increase connection timeout forlong running database operations

Reader/Writer

• Table selection

• Load data into KNIME

• Create table as select

• Insert/append data

• Delete rows from table

• Update values in table

Hive/Impala Loader

• Upload a KNIME data table to Hive/Impala

• Part of the commercial Big Data Extension

Hive/Impala Loader

Partitioning influencesperformance

Manipulation

• Filter rows and columns

• Join tables/queries

• Sort your data

• Write your own query

• Aggregate your data

Database GroupBy – DB Specific Aggregation Methods

SQLite 7 aggregation functions

PostgreSQL 25 aggregation functions

Database GroupBy – Aggregation Method Description

Database GroupBy – Manual Aggregation

Returns number of rows per group

Database GroupBy – Pattern Based Aggregation

Tick this option if the search pattern is a

regular expression otherwise it is treated

as string with wildcards ('*' and '?')

Database GroupBy – Type Based Aggregation

Matches all cells

Matches all numericcells

Database GroupBy – Custom Aggregation Function

Utility

• Drop table

– missing table handling

– cascade option

• Execute any SQL statement e.g. DDL

• Manipulate existing queries

Executes severalqueries separatedby ; and new line

In-Database Processing

Loads your pre-processeddata into KNIME

HDFS File Handling

• KNIME & Extensions -> KNIME File Handling Nodes

• HDFS Connection and HDFS File Permission nodes part of the commercial Big Data Extension

HDFS File Handling

Virtual Machines

• Hortonworks:

http://hortonworks.com/products/hortonworks-sandbox/

• Cloudera:http://www.cloudera.com/content/cloudera/en/downloads/quickstart_vms.html

• Virtual Box

https://www.virtualbox.org/

• VMWare Player

https://www.virtualbox.org/

Resources

• KNIME (www.knime.org)• BLOG for news, tips and tricks(www.knime.org/blog)

• FORUM for questions and answers (tech.knime.org/forum)

• EXAMPLE SERVER for example workflows

• LEARNING HUB (www.knime.org/learning-hub)

• KNIME TV channel on

• KNIME on @KNIME

• KNIME on

https://www.facebook.com/KNIMEanalytics

Harnessing Big Data with KNIME · Big Data, IoT, and the three V Variety: –KNIME inherently...

Documents

Transcript of Harnessing Big Data with KNIME · Big Data, IoT, and the three V Variety: –KNIME inherently...

KNIME Italy Meetup - Going Big Data on Apache Spark

The Open Analytics Platform - KNIME · 2017-11-07 · Webinars In 2018 Series of Webinars: •Data Science Cycle •Introduction to KNIME Analytics Platform •Data Access •Database

Big data for smart cities with KNIME a real experience in the ...

Big Data at your Desk with KNIME

Evotec - Knime Desktop Tools for Chemists - ChemAxon · Evotec AG, Knime –desktop tools for chemists, May 2011. PAGE Agenda 1 ... Knime is used to extract the data that has been

KNIME Big Data Workshop · 2019-03-29 · © 2018 KNIME.com AG. All Rights Reserved. KNIME Big Data Workshop Björn Lohrmann KNIME

Big Data Extensions Admin Guide - KNIME...Overview KNIME Big Data Extensions integrate Apache Spark and the Apache Hadoop ecosystem with KNIME Analytics Platform. This guide is aimed

Big Data Francesco Civardi - THE INNOVATION GROUP · Data • Collecting • ... • Vesenda eLegere, MSFT SQL Server, Analysis Services, PowerPivot and PowerViewer, RoamBI, KNIME,

Deploying KNIME to the Enterprise Reshaping Data ... · Deploying KNIME to the Enterprise Reshaping Data & Architecture for Healthcare. ... Extract data from multiple DB’s to ...

Big Data Analytics - KNIME · 2017-05-23 · Text Mining Data Mining Automation » Big Data Analytics Design and Optimization of Campaign Processes »Implementation of Campaign Management

KNIME - Create Workflow with KNIME

KNIME Server Workshop...Designed to integrate your advanced hardware and infrastructure capabilities for even better KNIME performance • KNIME Cluster Execution • KNIME Big Data

KNIME opens the Doors to Big Data...speed up the whole KNIME workflow execution by preparing the data quicker for further analytical processes. Once established that it would be beneficial

Schrodinger KNIME Extensionsgohom.win/.../knime/knime_user_manual.pdf · KNIME Extensions 1.5 User Manual 1 KNIME Extensions User Manual Chapter 1: Introduction 1.1 About KNIME KNIME,

Big Data Extensions Admin Guide - docs.knime.com · KNIME Big Data Extensions integrate Apache Spark and the Apache Hadoop ecosystem with KNIME Analytics Platform. This guide is aimed

KNIME & Avira, or how I‘ve learned to love Big Data...from Avira’s raw data within the dev. environment •Create machine-learning algorithms optimizing the offer (price, features)

KNIME Workbench Guide · KNIME Workbench After selecting a workspace for the current project, click Launch. The KNIME Analytics Platform user interface - the KNIME Workbench - opens.

Spring Summit 2017 - Welcome and Introduction - MB... · Scalability, Big Data, Cloud •Lots of Work on our Big Data Extensions –New&Cooking: KNIME on Spark, H20 Integration •Blending

Automated data analysis for HV batteries with KNIME...Automated data analysis for HV batteries with KNIME Maximilian Mücke (Deutsche ACCUmotive) and Patryk Koryzna (DATATRONIQ) Agenda

Data Science for Everyone - KNIME · 2017-05-23 · •The data scientist builds a data preparation and modeling workflow in KNIME capturing their most robust approach along with