Uwe Seiler, Data Architect and Trainer at codecentric AG - "Hadoop & Germany & 2016"

Hadoop & Germany & 2016

uweseiler

/whoami &

/disclaimer


We finally stopped talking infrastructure!


We now talk architectures and use cases!


#1 The Big Data Lake is an illusion!


Da

ta S

ourc

esD

ata

Sys

tem

sA

pp

lica

tion

s

Traditional Sources

RDBMS OLTP OLAP …

Traditional Systems

RDBMS EDW MPP …

Business Intelligence

BusinessApplications

Custom Applications

Operation

Manage &

Monitor

Dev Tools

Build &

Test

New Sources

Logs Mails Sensor …SocialMedia

EnterpriseHadoop Plattform

#1 The Vision of the Big Data Lake

Hadoop is not the one tool to rule them all

#1 Vision & Reality

Embrace heterogeneity! (and learn to deal with the complexity)

#1 After the reality shock…

#1 Real world architecture - Insurance

Da

ta S

ourc

esD

ata

Sys

tem

sA

pp

lica

tion

s

Traditional Sources

RDBMS OLTP OLAP …

Traditional Systems

DWH

BusinessIntelligence

New Sources

Logs Sensor …SocialMedia

Enterprise Hadoop Plattform

SAS LASR Server

Apache Zeppelin

#2 Speed is the new king!


#2 The “classic“ Lambda Architecture

Batch Layer

Speed LayerData Ingestion

Data Processing

Data Storage

Data Storage Data Analysis

Visualization

Visualization

…

DataChannels

ms - s

min - h

#2 Lambda in Action - (e)Commerce

SMACK Spark Mesos Akka

Cassandra Kafka

#2 The lust for speed

Data Ingestion

Data Processing

Raw Data

#2 Cassandra & Hadoop - AdServing

Data Processing

User Journey

Aggregated Data

Web Frontend

Aggregated Data< 120 days

Data Science

#3 Data Science to the help!


Hadoop is about to become commodity

#3 Let’s face it..

Algorithms will be the new differentiator

#3 We need new challenges…

Batch Layer

Speed LayerData Ingestion

Stream Processing

ms - s

min - h

#3 Fraud detection - Financial services

DataImport

Data Preparation

Model Generation

Model Validation

Feature & Parameter Selection

Manual or automatic Iterations to tune

parameters

Use Model

Refresh Model from latest input data

Every major company is building teams of unicorns

#3 The solution?

#4 Hadoop for good!


Hadoop User Group Rhein-Mainhttp://www.meetup.com/de-DE/HUG-Rhein-Main/

Next Meetup: 23.06.2016, Talks welcome

Uwe Seiler, Data Architect and Trainer at codecentric AG - "Hadoop & Germany & 2016"

Technology

Transcript of Uwe Seiler, Data Architect and Trainer at codecentric AG - "Hadoop & Germany & 2016"