Introducing Amazon Kinesis: Real-time Processing of Streaming Big Data (BDT103) | AWS re:Invent 2013
Extracting Real-Time Insights from Streaming Data · Dilip Kumar, VP of Technology for Amazon Go,...
75
Extracting Real-Time Insights from Streaming Data Roger Barga, GM Robotics & Autonomous Systems Amazon Web Services
Transcript of Extracting Real-Time Insights from Streaming Data · Dilip Kumar, VP of Technology for Amazon Go,...
Extracting Real-Time Insights from Streaming Data
Roger Barga, GM
Robotics & Autonomous Systems
Amazon Web Services
#Real-time
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
• Durable
• Continuous
• Fast
• Correct
• Reactive
• Reliable
Process Real-Time Streaming Data
Key Requirements?
Ingest Transform Analyze React Persist
Amazon CloudWatch Logs
Amazon S3 Events
AWS Metering Amazon.com’s Catalog
We All Use Kinesis
At Hearst, the Clickstream is the real-timetransmission, collection, and processing of thedata generated by Hurst customers across all theirdevices
Action Data
Scrolling or Mouse movement
Event Data
Highlighting Text, Listening to Audio, Playing Video
Geospatial Data
Lat/Lon, GPS, Movement, Proximity
Sensor Data
Pulse, Gait, Body Temperature
What business is Hearst in?
Magazines20 U.S. titles & nearly 300 international titles
What business is Hearst in?
Newspapers15 daily & 34 weekly titles
What business is Hearst in?
Broadcasting30 television & 2 radio stations
What business is Hearst in?
Business MediaOperates more than 20 business-to businesses with significant holdings in the auto, electronic, medical
and financial industries
What business is Hearst in?
Hearst has over 300 websites world-wide, which results in 1TB of data per day and over 20 billion
pageviews per year.
What business is Hearst in?
“Hearst is in the Data Creation Business”
Buzzing@Hearst
Business Value
Real-Time Reactions Instant feedback on articles from their audiences
Promoting Popular Content Cross-ChannelIncremental re-syndication of popular articles across properties(e.g. trending newspaper articles can be adopted by magazines)
Authentic InfluenceInform Hearst editors to write articles that are more relevant to their audiences
Understanding EngagementInform both editors what channels their audiences are leveraging to read Hearst articles
INCREMENTAL REVENUE
25% more page views
15% more visitors
Customer Examples
1 billion events per
week from
connected devices
Near-real-time
home valuation
(Zestimates)
Live clickstream
dashboards refreshed
under 10ms
100 GB/day
clickstreams from
250+ sites
50 billion daily ad
impressions, sub-
50 ms responses
Online stylist
processing 10
million events/day
Migrated data bus
from Kafka to
Kinesis
Facilitate
communications
between 100+
microservices
Analyze billions of
network flows in
real-time
Real-time home
recommendations
Stream New Data in Seconds Get actionable insights quickly
Streaming
Ingest video & data as it’s generated
Process data on the fly
Real-timeAnalytics/ML, alerts, actions
Dilip Kumar, VP of Technology for Amazon Go, cites as an example Amazon Go’s unique system of streaming data from hundreds of cameras to track the shopping activities of customers. The innovations his team concocted helped influence an AWS service called Kinesis, which allows customers to stream video from multiple devices to the Amazon cloud, where they can process it, analyze it, and use it to further advance their machine learning efforts.
Speed Matters
Kinesis Streaming Services
Robust Random Cut ForrestSummary of a dynamic data stream, highly efficient, wide number of use cases…
Random Cut Tree
Recurse: The cutting stops when each point is isolated.
Range-biased Cut
0 10
5
Random Cut Forest
Each tree built on a random sample.
…
Random Sample of a Stream
Reservoir Sampling [Vitter]
Maintain random sample of 5 points in a stream?
Keep heads with probability
Discard tails with probability
56
16
Insert – Case I
Start with the Root
If the point falls inside the bounding box
follow the path to the appropriate child
Insert – Case II
Theorem: Insert generates a tree T’ ~ T( )
What is an Outlier?
Anomaly Score: Displacement
A point is an anomaly if its insertion greatly increases the tree size ( = sum of path lengths from root to leaves
= description length).
Inlier:
Anomaly Score: Displacement
Outlier
NYC Taxi Ridership
0
5000
10000
15000
20000
25000
30000
20
14-1
2-0
1 0
0:0
0:0
0
20
14-1
2-0
1 0
4:0
0:0
0
20
14-1
2-0
1 0
8:0
0:0
0
20
14-1
2-0
1 1
2:0
0:0
0
20
14-1
2-0
1 1
6:0
0:0
0
20
14-1
2-0
1 2
0:0
0:0
0
20
14-1
2-0
2 0
0:0
0:0
0
20
14-1
2-0
2 0
4:0
0:0
0
20
14-1
2-0
2 0
8:0
0:0
0
20
14-1
2-0
2 1
2:0
0:0
0
20
14-1
2-0
2 1
6:0
0:0
0
20
14-1
2-0
2 2
0:0
0:0
0
20
14-1
2-0
3 0
0:0
0:0
0
20
14-1
2-0
3 0
4:0
0:0
0
20
14-1
2-0
3 0
8:0
0:0
0
20
14-1
2-0
3 1
2:0
0:0
0
20
14-1
2-0
3 1
6:0
0:0
0
20
14-1
2-0
3 2
0:0
0:0
0
20
14-1
2-0
4 0
0:0
0:0
0
20
14-1
2-0
4 0
4:0
0:0
0
20
14-1
2-0
4 0
8:0
0:0
0
20
14-1
2-0
4 1
2:0
0:0
0
20
14-1
2-0
4 1
6:0
0:0
0
20
14-1
2-0
4 2
0:0
0:0
0
20
14-1
2-0
5 0
0:0
0:0
0
20
14-1
2-0
5 0
4:0
0:0
0
20
14-1
2-0
5 0
8:0
0:0
0
20
14-1
2-0
5 1
2:0
0:0
0
20
14-1
2-0
5 1
6:0
0:0
0
20
14-1
2-0
5 2
0:0
0:0
0
20
14-1
2-0
6 0
0:0
0:0
0
20
14-1
2-0
6 0
4:0
0:0
0
20
14-1
2-0
6 0
8:0
0:0
0
20
14-1
2-0
6 1
2:0
0:0
0
20
14-1
2-0
6 1
6:0
0:0
0
20
14-1
2-0
6 2
0:0
0:0
0
20
14-1
2-0
7 0
0:0
0:0
0
20
14-1
2-0
7 0
4:0
0:0
0
20
14-1
2-0
7 0
8:0
0:0
0
20
14-1
2-0
7 1
2:0
0:0
0
20
14-1
2-0
7 1
6:0
0:0
0
20
14-1
2-0
7 2
0:0
0:0
0
numPassengers
Mon
8am
6pm
4pm
Sat
11pm11am
Tue Wed Thu Fri
Date aggregated every 30 minutes,Shingle Size: 48
1Public Data: http://www.nyc.gov/html/tlc/html/about/trip_record_data.shtml
NYC Taxi Data
0
5000
10000
15000
20000
25000
30000
35000
40000
45000
20
14-0
9-1
6 2
1:3
0:0
020
14-0
9-1
9 1
3:0
0:0
020
14-0
9-2
2 0
4:3
0:0
020
14-0
9-2
4 2
0:0
0:0
020
14-0
9-2
7 1
1:3
0:0
020
14-0
9-3
0 0
3:0
0:0
020
14-1
0-0
2 1
8:3
0:0
020
14-1
0-0
5 1
0:0
0:0
020
14-1
0-0
8 0
1:3
0:0
020
14-1
0-1
0 1
7:0
0:0
020
14-1
0-1
3 0
8:3
0:0
020
14-1
0-1
6 0
0:0
0:0
020
14-1
0-1
8 1
5:3
0:0
020
14-1
0-2
1 0
7:0
0:0
020
14-1
0-2
3 2
2:3
0:0
020
14-1
0-2
6 1
4:0
0:0
020
14-1
0-2
9 0
5:3
0:0
020
14-1
0-3
1 2
1:0
0:0
020
14-1
1-0
3 1
2:3
0:0
020
14-1
1-0
6 0
4:0
0:0
020
14-1
1-0
8 1
9:3
0:0
020
14-1
1-1
1 1
1:0
0:0
020
14-1
1-1
4 0
2:3
0:0
020
14-1
1-1
6 1
8:0
0:0
020
14-1
1-1
9 0
9:3
0:0
020
14-1
1-2
2 0
1:0
0:0
020
14-1
1-2
4 1
6:3
0:0
020
14-1
1-2
7 0
8:0
0:0
020
14-1
1-2
9 2
3:3
0:0
020
14-1
2-0
2 1
5:0
0:0
020
14-1
2-0
5 0
6:3
0:0
020
14-1
2-0
7 2
2:0
0:0
020
14-1
2-1
0 1
3:3
0:0
020
14-1
2-1
3 0
5:0
0:0
020
14-1
2-1
5 2
0:3
0:0
020
14-1
2-1
8 1
2:0
0:0
020
14-1
2-2
1 0
3:3
0:0
020
14-1
2-2
3 1
9:0
0:0
020
14-1
2-2
6 1
0:3
0:0
020
14-1
2-2
9 0
2:0
0:0
020
14-1
2-3
1 1
7:3
0:0
020
15-0
1-0
3 0
9:0
0:0
020
15-0
1-0
6 0
0:3
0:0
020
15-0
1-0
8 1
6:0
0:0
020
15-0
1-1
1 0
7:3
0:0
020
15-0
1-1
3 2
3:0
0:0
020
15-0
1-1
6 1
4:3
0:0
020
15-0
1-1
9 0
6:0
0:0
020
15-0
1-2
1 2
1:3
0:0
020
15-0
1-2
4 1
3:0
0:0
020
15-0
1-2
7 0
4:3
0:0
020
15-0
1-2
9 2
0:0
0:0
0
numPassengers
NYC Taxi Data
0
10
20
30
40
50
60
70
80
90
100
0
5000
10000
15000
20000
25000
30000
35000
40000
45000
20
14-0
9-1
6 2
1:3
0:0
020
14-0
9-1
9 1
5:3
0:0
020
14-0
9-2
2 0
9:3
0:0
020
14-0
9-2
5 0
3:3
0:0
020
14-0
9-2
7 2
1:3
0:0
020
14-0
9-3
0 1
5:3
0:0
020
14-1
0-0
3 0
9:3
0:0
020
14-1
0-0
6 0
3:3
0:0
020
14-1
0-0
8 2
1:3
0:0
020
14-1
0-1
1 1
5:3
0:0
020
14-1
0-1
4 0
9:3
0:0
020
14-1
0-1
7 0
3:3
0:0
020
14-1
0-1
9 2
1:3
0:0
020
14-1
0-2
2 1
5:3
0:0
020
14-1
0-2
5 0
9:3
0:0
020
14-1
0-2
8 0
3:3
0:0
020
14-1
0-3
0 2
1:3
0:0
020
14-1
1-0
2 1
5:3
0:0
020
14-1
1-0
5 0
9:3
0:0
020
14-1
1-0
8 0
3:3
0:0
020
14-1
1-1
0 2
1:3
0:0
020
14-1
1-1
3 1
5:3
0:0
020
14-1
1-1
6 0
9:3
0:0
020
14-1
1-1
9 0
3:3
0:0
020
14-1
1-2
1 2
1:3
0:0
020
14-1
1-2
4 1
5:3
0:0
020
14-1
1-2
7 0
9:3
0:0
020
14-1
1-3
0 0
3:3
0:0
020
14-1
2-0
2 2
1:3
0:0
020
14-1
2-0
5 1
5:3
0:0
020
14-1
2-0
8 0
9:3
0:0
020
14-1
2-1
1 0
3:3
0:0
020
14-1
2-1
3 2
1:3
0:0
020
14-1
2-1
6 1
5:3
0:0
020
14-1
2-1
9 0
9:3
0:0
020
14-1
2-2
2 0
3:3
0:0
020
14-1
2-2
4 2
1:3
0:0
020
14-1
2-2
7 1
5:3
0:0
020
14-1
2-3
0 0
9:3
0:0
020
15-0
1-0
2 0
3:3
0:0
020
15-0
1-0
4 2
1:3
0:0
020
15-0
1-0
7 1
5:3
0:0
020
15-0
1-1
0 0
9:3
0:0
020
15-0
1-1
3 0
3:3
0:0
020
15-0
1-1
5 2
1:3
0:0
020
15-0
1-1
8 1
5:3
0:0
020
15-0
1-2
1 0
9:3
0:0
020
15-0
1-2
4 0
3:3
0:0
020
15-0
1-2
6 2
1:3
0:0
020
15-0
1-2
9 1
5:3
0:0
0
numPassengers Anomaly Score
NYC Marathon
Christmas
New Years
Snowstorm
Robust Random Cut ForestsQuick Summary
• Random forests (RF) define ensemble models;• The cut in RCF corresponds to specific choice of partitioning;
• Take a set of points, compute bounding box; • Choose an axis proportional to the length (biased), then choose an
uniform random cut in range;• Recurse on both sides.
• Isolation Forests: Partition at random, dimensions unbiased;• Why RCFs? Can maintain RCF tree distributions efficiently and do
so online as data is streaming in. Distance preserving.
Proceedings 33rd International Conference on Machine Learning, New York, NY, USA, 2016.
What we are going to cover…
Random Cut Forests
Anomaly Detection
Density Estimation Forecasting
Semi-Supervised Learning
Attribution & Directionality
Explainable/Transparent/Interpretable ML
“If my time-series data with 30 features yields an unusually high anomaly score. How do I explain why this particular point in the time- series is unusual? [..] Ideally I’m looking for some way to visualize “feature importance” for a specific data point.”
--- Robin Meehan, Inasight.com
What is Attribution?
It’s the ratio of the “distance” of the anomaly from normal.
(It’s a distance in space of repeated patterns in the data.)
What is Attribution?
An
om
aly Sco
re
NYC Taxi Ridership Data1
1Public Data: http://www.nyc.gov/html/tlc/html/about/trip_record_data.shtml
An
om
aly
Sco
re
Explaining Anomalies
The Moving Example
A Fan/Turbine
1000 pts in each blade
Gaussian, for simplicity
Blades designed unequal
Rotate counterclockwise
3 special “query” points
100 trees, 256 points each
Anomaly Score at P1
What is going on at 90 degrees?
Blade overhead = Not an anomaly
All 3 Blades
Attribution
p1 is far away in x-coord most of the time
But what is happening to y?
x coordinate’s contribution for p1?
Directionality
Sharp transition when the blademoves from above to below at p1!Total score plummets.
Slowly rotating awayTotal score remains high
Forecasting
Exhibit B: Forecasting using RCFs
Not just the next value!
Ability to “see past” anomalies
Auto-detect periodicity …
Forecasting implies missing value imputation!
Wait, this is just a sine wave … its easy ...
Maybe not…
Realistic Data?
ECG (one lead)
Periodicity unclear …
Shingle Size = 185
Test on hold out data
Paper in preparation!(Sudipto Guha)
Semi-Supervised Learning on Data Streams
One Hour
Amazon.com Orders per Minute
In a long stream…
Proceedings of the 35th International Conference on Machine Learning, Stockholm, Sweden, PMLR 80, 2018.
What we saw today…
Random Cut Forests
Anomaly Detection
Density Estimation Forecasting
Semi-Supervised Learning
Attribution & Directionality
Detecting Anomalies in Streaming Graphs
Contributors To This Project
Roger Barga, Dhivya Eswaran, Gaurav Ghare, Kapil Chhabra, Sudipto Guha, Shiva Kasiviswanathan, Nina Mishra,
Gourav Roy, Joshua Tokle, and Tal Wagner
