Tapjoy: Building a Real-Time Data Science Service for Mobile Advertising
-
Upload
memsql -
Category
Technology
-
view
152 -
download
2
Transcript of Tapjoy: Building a Real-Time Data Science Service for Mobile Advertising
![Page 1: Tapjoy: Building a Real-Time Data Science Service for Mobile Advertising](https://reader036.fdocuments.net/reader036/viewer/2022062523/58e4a1621a28aba3458b60c7/html5/thumbnails/1.jpg)
Building Real-timeAnalytics Engine with Kafka and Spark for Mobile Advertising
![Page 2: Tapjoy: Building a Real-Time Data Science Service for Mobile Advertising](https://reader036.fdocuments.net/reader036/viewer/2022062523/58e4a1621a28aba3458b60c7/html5/thumbnails/2.jpg)
Mobile Advertising? - Social & Game
Authentic to Consumers Authentic to Entertainment
Authentic to Engagement
Mobile Games
![Page 3: Tapjoy: Building a Real-Time Data Science Service for Mobile Advertising](https://reader036.fdocuments.net/reader036/viewer/2022062523/58e4a1621a28aba3458b60c7/html5/thumbnails/3.jpg)
eMarketer, “US Mobile Phone Content Usage Metrics, 2013-2019.” February, 2015.
People Spend A Lot of Time Gaming
3
Over 55 minutes a day on average is spent playing mobile games
Minutes Spent in Mobile
eMarketer, “US Mobile Phone Content Usage Metrics, 2013-2019.” February, 2015.
![Page 4: Tapjoy: Building a Real-Time Data Science Service for Mobile Advertising](https://reader036.fdocuments.net/reader036/viewer/2022062523/58e4a1621a28aba3458b60c7/html5/thumbnails/4.jpg)
Innovate Advertising as Reward Ads
● Free-to-Play (Freemium) App● Only 2~5% users In-app-purchase● Publisher can give “reward” on users who engaged to Ads● Video + Game Economics + Reward
![Page 5: Tapjoy: Building a Real-Time Data Science Service for Mobile Advertising](https://reader036.fdocuments.net/reader036/viewer/2022062523/58e4a1621a28aba3458b60c7/html5/thumbnails/5.jpg)
Mobile Video App Advertising
AdvertiserPay on Video-View
Pub Paid
Tapjoy Profit
User Earn Reward
![Page 6: Tapjoy: Building a Real-Time Data Science Service for Mobile Advertising](https://reader036.fdocuments.net/reader036/viewer/2022062523/58e4a1621a28aba3458b60c7/html5/thumbnails/6.jpg)
Video to Install
Video Install
reward No reward
![Page 7: Tapjoy: Building a Real-Time Data Science Service for Mobile Advertising](https://reader036.fdocuments.net/reader036/viewer/2022062523/58e4a1621a28aba3458b60c7/html5/thumbnails/7.jpg)
Video to Install to Event
Video Install
reward
No reward
Event
- Level N- Registration- In-app-purchase- First Booking
![Page 8: Tapjoy: Building a Real-Time Data Science Service for Mobile Advertising](https://reader036.fdocuments.net/reader036/viewer/2022062523/58e4a1621a28aba3458b60c7/html5/thumbnails/8.jpg)
Mobile Video App Advertising - Data Science
Video Views
Installs
Early Retention
Life Time Value
“Event”
Look-alike Model
Real-timeBidding Engine
Advertiser’s Return
“Investment”
![Page 9: Tapjoy: Building a Real-Time Data Science Service for Mobile Advertising](https://reader036.fdocuments.net/reader036/viewer/2022062523/58e4a1621a28aba3458b60c7/html5/thumbnails/9.jpg)
Building a Data Science Platform
Bigger in Scale
FasterServin
g
Smart and
Smarter !
Data Product
![Page 10: Tapjoy: Building a Real-Time Data Science Service for Mobile Advertising](https://reader036.fdocuments.net/reader036/viewer/2022062523/58e4a1621a28aba3458b60c7/html5/thumbnails/10.jpg)
Tapjoy’s Data Platform
Algo Serving InfrastructureDatawarehousing
300,000 RPM throughput
Bidding & Targeting &
Personalization
<10 ms response time
20 TB daily addition2.3 PB DUM
Cloud & On-PremiseIn-house & SaaSBatch based & real-time
![Page 11: Tapjoy: Building a Real-Time Data Science Service for Mobile Advertising](https://reader036.fdocuments.net/reader036/viewer/2022062523/58e4a1621a28aba3458b60c7/html5/thumbnails/11.jpg)
The Logic Stack
Data warehousing
HDFS / S3 / GSReporting
MPPs (BigQuery)
Algo Service
Batch + Streaming Hadoop / Spark
• Collect data, set rules• Reduce data friction• Improve signal-to-noise
ratio• Model training & iteration
• Deliver business insights• Driving data awareness
• Apply ideas to product (online)
• Serve model output• Drive revenue
Data Viz
A/B TestingData Viz
![Page 12: Tapjoy: Building a Real-Time Data Science Service for Mobile Advertising](https://reader036.fdocuments.net/reader036/viewer/2022062523/58e4a1621a28aba3458b60c7/html5/thumbnails/12.jpg)
The Data Flow
![Page 13: Tapjoy: Building a Real-Time Data Science Service for Mobile Advertising](https://reader036.fdocuments.net/reader036/viewer/2022062523/58e4a1621a28aba3458b60c7/html5/thumbnails/13.jpg)
Tapjoy’s Algo Service Engine (SOA)
● SOA (algo service) in Natty● 320, 000 lines of Java● 99% response time < 20 ms @ 200k - 400k RPM
Ad Request
A/B test classification
Main Algo & pre-filters
Apply Logic Pipe
Response (offer list)
Video BiddingTargetingPersonaLookalike
...
Biz logic filters
![Page 14: Tapjoy: Building a Real-Time Data Science Service for Mobile Advertising](https://reader036.fdocuments.net/reader036/viewer/2022062523/58e4a1621a28aba3458b60c7/html5/thumbnails/14.jpg)
Algo Service’s Data Components
Component What’s in there Purpose
Kafka Raw activity logs Everything starts here
Spark Streaming ETL ETL & Algo feature updates
Aerospike User Big Table (User DNA) Real-time k-v lookups. I.e LookALike
MemSQL Striped down raw user activity data!!
● Device level real time aggregations
● Hot data sink ● Real time reporting
Elasticsearch Aggregates or Unstructured logs
Cube aggregates or fulltext search
![Page 15: Tapjoy: Building a Real-Time Data Science Service for Mobile Advertising](https://reader036.fdocuments.net/reader036/viewer/2022062523/58e4a1621a28aba3458b60c7/html5/thumbnails/15.jpg)
Mobile Video App Advertising - Data Science
Video Views
Installs
Early Retention
Life Time Value
“event”
Look-alike Model
Real-timeBidding Engine
Advertiser’s Return
“Investment”
![Page 16: Tapjoy: Building a Real-Time Data Science Service for Mobile Advertising](https://reader036.fdocuments.net/reader036/viewer/2022062523/58e4a1621a28aba3458b60c7/html5/thumbnails/16.jpg)
Big Table / MemCache
Use Case 1 - Ad-Request Level Decision
Video Bid
# CVR
Spending History
max(views) > T(n)
...
User app usages
Kafka+
Spark Streaming
S - App 1
S - App 2
S - App 3
S - App ..
S - App N
Lamda Batch
![Page 17: Tapjoy: Building a Real-Time Data Science Service for Mobile Advertising](https://reader036.fdocuments.net/reader036/viewer/2022062523/58e4a1621a28aba3458b60c7/html5/thumbnails/17.jpg)
Use Case 1 - Ad-Request Level Decision
Video BidKafka
ORSpark
Streaming
S - App
RAW DATA
![Page 18: Tapjoy: Building a Real-Time Data Science Service for Mobile Advertising](https://reader036.fdocuments.net/reader036/viewer/2022062523/58e4a1621a28aba3458b60c7/html5/thumbnails/18.jpg)
Use Case 1 - Ad-Request Level Decision High throughput low latency queries querying 30 days device
level data which are streamed into MemSQL.
Does the calculations on the fly and serving as decision features
Reference Join Subquery
Reference Join
![Page 19: Tapjoy: Building a Real-Time Data Science Service for Mobile Advertising](https://reader036.fdocuments.net/reader036/viewer/2022062523/58e4a1621a28aba3458b60c7/html5/thumbnails/19.jpg)
In Fact - One Fits All
Algo Serving
KafkaOR
Spark Streaming
Real-Time Dashboard
Data Warehouse Hot Batch
Data Sink
HotBatch
RealtimeQuery
RealtimeQuery
![Page 20: Tapjoy: Building a Real-Time Data Science Service for Mobile Advertising](https://reader036.fdocuments.net/reader036/viewer/2022062523/58e4a1621a28aba3458b60c7/html5/thumbnails/20.jpg)
eMarketer, “US Mobile Phone Content Usage Metrics, 2013-2019.” February, 2015.
Conclusion
20
❖ Mobile Advertising is all about knowing your audience❖ Fast & Accurate data is key to Data Science as Service❖ But, “Realtime” is a relative word❖ Try to simplify moving parts when it come to streaming
➢ Difficult to debug➢ Hard to backfill
❖ Generalized hot-data sink for stability and multi-purpose data storage
[email protected]@tapjoy.com