Beyond Scaling: Real-Time Event Processing with Stream Mining - … · 2013. 6. 12. · Mikio Braun...

Mikio Braun Beyond Scaling: Real-Time Event Processing with Stream Mining Berlin Buzzwords 2013 © by MB

Beyond Scaling: Real-Time Event Processing with

Stream Mining

Mikio L. Braun, @mikiobraunhttp://blog.mikiobraun.de

TWIMPACThttp://twimpact.com

Event Data

Attribution: flickr users kenteegardin, fguillen, torkildr, Docklandsboy, brewbooks, ellbrown, JasonAHowie

FinanceFinance GamingGaming MonitoringMonitoring

Social MediaSocial MediaSensor NetworksSensor NetworksAdvertismentAdvertisment

Event Data is Huge: Volume

● The problem: You easily get A LOT OF DATA!– 100 events per second

– 360k events per hour

– 8.6M events per day

– 260M events per month

– 3.2B events per year

Event Data is Huge: Diversity

● Potentially large spaces:– distinct words: >100k

– IP addresses: >100M

– users in a social network: >10M

http://www.flickr.com/photos/arenamontanus/269158554/http://wordle.net

To Scale Or Not To Scale?

Stream Mining to the rescue

● Stream mining algorithms:– answer “stream queries” with finite resources

● Typical examples:– how often does an item appear in a stream?

– how many distinct elements are in the stream?

– what are the top-k most frequent items?

Continuous Stream of Data

Bounded ResourceAnalyzer Stream Queries

The Trade-Off

ExactFast

Big Data

Stream Mining Map Reduce and friends

First seen here: http://www.slideshare.net/acunu/realtime-analytics-with-apache-cassandra

Heavy Hitters (a.k.a. Top-k)

● Count activities over large item sets (millions, even more, e.g. IP addresses, Twitter users)

● Interested in most active elements only.

Metwally, Agrawal, Abbadi, Efficient computation of Frequent and Top-k Elements in Data Streams, Internation Conference on Database Theory, 2005

Fixed tables of counts

Case 1: element already in data base

Case 2: new element

142 142 12 13

713 023 2

Heavy Hitters over Time-Window

● Keep quite a big log (a month?)

● Constant write/erase in database

● Alternative: Exponential decay

Exponential Decay

● Instead of a fixed window, use exponential decay

● The beauty: updates are recursivehalftime

timestamp

time shift term

Exponential Decay

● Collect stats by a table of expdecay counterscounters[item] # counters

ts[item] # last timestamp

● update(C, item, timestamp, count) – update countsC.counters[item] = count + weight(timestamp, C.ts[item]) * C.counters[item]

C.ts[item] = timestampC.lastupdate = timestamp

● score(C, item) – return scorereturn weight(C.lastupdate, C.ts[item]) * C.counters[item]

Count-Min Sketches

● Summarize histograms over large feature sets● Like bloom filters, but better

● Query: Take minimum over all hash functions

0 0 3 0

1 1 0 2

0 2 0 0

0 3 5 2

0 5 3 2

2 4 5 0

1 3 7 3

0 2 0 8

m bins

n differenthash functions

Updates for new entryQuery result: 1

G. Cormode and S. Muthukrishnan. An improved data stream summary: The count-min sketch and its applications. LATIN 2004, J. Algorithm 55(1): 58-75 (2005) .

Wait a minute? Only Counting?

● Well, getting the top most active items is already useful.– Web analytics, Users, Trending Topics

● Counting is statistics!

Counting is Statistics

● Empirical mean:

● Correlations:

● Principal Component Analysis:

Example 1: Least Squares Regression

with entries

this could be huge!

d x d is probably ok

Idea: Batch method like least squares on recent portion of the data.

But: It's just a sum!

Least Squares Regression

● Need to compute● For each do

● Then, reconstruct –

As a reminder:

Example 2: Maximum-Likelihood

● Estimate probabilistic models

● But wait, how do I “1/n” with randomly spaced events?

based on

which is slightly biased, but simpler

Example 3: Outlier detection

● Once you have a model, you can compute p-values (based on recent time frames!)

Example 4: Clustering● Online clustering

– For each data point:● Map to closest centroid ( compute distances)⇒● Update centroid

– count-min sketches to represent sum over all vectors in a class

Aggarwal, A Framework for Clustering Massive-Domain Data Streams, IEEE International Conference on Data Engineering , 2009

0 0 3 01 1 0 2

0 2 0 00 3 5 2

0 5 3 22 4 5 0

1 3 7 30 2 0 8

Example 5: TF-IDF

● estimate word – document frequencies

● for each word: update(word, t, 1.0)● for each document: update(“#docs”, t, 1.0)● query: score(word) / score(“#docs”)

Example 6: Classification with Naïve Bayes● Naive Bayes is also just counting, right?

class priors

Multinomnial naïve Bayes

Priors

Total number of words in class

Number of times word appears in classfrequency of word in document

Example 6: Classification with Naive Bayes

ICML 2003

Example 6: Classification with Naive Bayes● 7 Steps to improve NB:

– transform TF to log( . + 1)

– IDF-style normalization

– square length normalization

– use complement probability

– another log

– normalize those weights again

– Predict linearly using those weights

What about non-parametric methods and Kernel Methods?● Problem here, no real accumulation of information

in statistics, e.g. SVMs

● Could still use streamdrill to extract a representative subset.

sum over all elements!

Streamdrill

● Heavy Hitters counting + exponential decay● Instant counts & top-k results over time

windows.● Indices!● Snapshots for historical analysis● Beta demo available at http://streamdrill.com,

launch imminent

Architecture Overview

Example: Twitter Stock Analysis

http://play.streamdrill.com/vis/

Summary

● Doesn't always have to be scaling!● Stream mining: Approximate results with

finite resources.● streamdrill: stream analysis engine

Beyond Scaling: Real-Time Event Processing with Stream Mining - … · 2013. 6. 12. · Mikio Braun...

Documents

Transcript of Beyond Scaling: Real-Time Event Processing with Stream Mining - … · 2013. 6. 12. · Mikio Braun...

Projektgruppe 542 Stream Mining for Intrusion … · Projektgruppe 542 Stream Mining for Intrusion Detection Zwischenbericht 27. September 2010 Matthias Balke Tobias Beckers Klaudia

Mobile Data Stream Mining (Advances)

Scaling big-data-mining-infra2

Beyond Scaling Real Time Even Processing With Stream Mining

Scaling Big Data Mining Infrastructure: The Twitter Experiencelintool.github.io/UMD-courses/bigdata-2013-Spring/... · 2016-01-29 · Scaling Big Data Mining Infrastructure: The Twitter

In-stream gravel mining impacts and environmental degradation

Adaptive, Hands-Oﬀ Stream Mining - bitquill.netbitquill.net/pdf/stream_vldb03.pdf · Adaptive, Hands-Oﬀ Stream Mining ... Pittsburgh, PA, USA {spapadim,christos}@cs.cmu.edu, abrock@stat.cmu.edu

Auto-scaling Techniques for Elastic Data Stream Processing · Auto-scaling Techniques for Elastic Data Stream Processing Thomas Heinze, ... Auto-scaling Techniques for Elastic Data

Data Stream Mining

More Stream-Mining

Data Mining of Informational Stream in Social Networks

Context-Adaptive Big Data Stream Mining - Online Appendix

Scaling up data mining algorithms: review and taxonomy · with this problem: scaling up data mining algorithms and data reduction. However, because data reduction is a data mining

Scaling Big Data Mining Infrastructure: The Twitter Experience › exploration_files › V14-02-02-Lin.pdf · Scaling Big Data Mining Infrastructure: The Twitter Experience Jimmy

BigData Stream Mining

Stream Hierarchy Data Mining for Sensor Data

EdgeScaler: Effective Elastic Scaling for Graph Stream ...Graph Processing, Stream Processing, Elastic Scaling ACM Reference Format: Daniel Presser, Frank Siqueira, Luís Rodrigues,

Handling concept drift in data stream mining

Realtime Personalization and Recommendation with Stream Mining

Introduction to data stream querying and mining · Page 11 G.HEBRAIL – May 5-78, 2009 Introduction to data stream querying and mining Applications of data stream processing Network