Streetlife's real time analytics stack
37
ANALYTICS
-
Upload
idanby -
Category
Technology
-
view
160 -
download
1
Transcript of Streetlife's real time analytics stack
ANALYTICS
• Local social network • Web, iOS and Android • 1.3M users in uk. • 1.5B events
analytics are core for our business
we track millions of events a day
Subject: Here's what your neighbours are talking about... To: [email protected]
Subject: Here's what your neighbours are talking about... To: [email protected]
Subject: Here's what your neighbours are talking about... To: [email protected]
Susan P.
Susan P.
Susan P.
we currently use the batch pipeline
Redshift
we gather insights once a day
Redshift
user_stats
message_statsSQL
last_visit last_email_open
2016/01/15 12:33:54
2016/07/12 22:13:10
2016/07/10 04:52:01
2016/03/11 21:01:03
2016/07/11 05:34:45
2016/07/12 14:30:21
day impressions
2016/01/15
engagmentsid
1
u_impressions u_engagments
345 122 299 99
2016/01/152 123 99 94 84
2016/01/153 934 845 843 789
2016/01/141 899 777 744 645
2016/01/142 754 543 634 433
2016/01/143 103 99 91 77
2016/01/131 499 301 382 235
2016/01/132 1893 1400 991 1099
With this data we give businesses insights
Your message got 890 impressions and 350 engagements.
We tune each message's reach dynamically
2000
target
1200
current
>
It works great, but the data has several hours of lag
event
ingest
Load SQL View S3 Load
We are going to start using the realtime pipeline
realtime analytics
realtime content router
realtime pipeline
Redshift
Kinesis
λ
Kinesis
λ λ
Dynamodb
LambdaLambda
The latency from events happening to us acting on them would be nearly zero
event
ingest
Kinesis Lambda DynamoDB Load
AWS Lambdas: 20,000 feet view
.py
.js
.java
DEPENDENCIES
ZIP
AWSS3λ
λ • Zero-administration compute platform • Connect Lambdas with AWS services
• Kinesis, DynamoDB, APIGateway, S3, CloudWatch, Cron…
• Pricing based on usage: • ~$0.0000002 per/run
• ~$0.000000208 per/100ms
λλ
AWS Lambdas: 20,000 feet view
There is a ‘but’ Deploying/Managing them is a pitaλ
• Tool to create, wire and deploy AWS Lambdas using CloudFormation
• Python/Javascript/Java/Golang/Scala runtimes… • Supported integrations
• APIGateway, CloudWatch, Dynamodb, Kinesis, S3
+
snowplow events
simulator
λ DynamoDB
Kinesis
We are hiring!
Thank you!
Software Engineer (data team) Software Engineer (core team)
Infrastructure Engineer Data Scientist
