Shimon Tolts, R&D Manager, IronSource - "Data Flow Management"
-
Upload
dataconomy-media -
Category
Technology
-
view
118 -
download
0
Transcript of Shimon Tolts, R&D Manager, IronSource - "Data Flow Management"
![Page 1: Shimon Tolts, R&D Manager, IronSource - "Data Flow Management"](https://reader036.fdocuments.net/reader036/viewer/2022070516/58713b181a28abf0568b6d21/html5/thumbnails/1.jpg)
Shimon Tolts General Manager, Data Solutions
ironSource Atom
Data Flow Management
![Page 2: Shimon Tolts, R&D Manager, IronSource - "Data Flow Management"](https://reader036.fdocuments.net/reader036/viewer/2022070516/58713b181a28abf0568b6d21/html5/thumbnails/2.jpg)
About ironSource: ironSource in Numbers
700Employees
Sep. 2010
Established
50%R&D Employees
700Advertisers
80KPartnered Apps
100MDevices using ironSource
solutions shipping in 2016
TEL AVIV ISRAEL
SAN FRANCISCO UNITED STATES
NEW YORK UNITED STATES
LONDON UNITED KINGDOM
BANGALORE INDIA
HONG KONG CHINA
KIEV UKRAINE
BEIJING CHINA SHANGHAI CHINA
![Page 3: Shimon Tolts, R&D Manager, IronSource - "Data Flow Management"](https://reader036.fdocuments.net/reader036/viewer/2022070516/58713b181a28abf0568b6d21/html5/thumbnails/3.jpg)
About ironSource: ironSource Hypergrowth
People Reached Each Month
4200Apps Installed Every Minutewith the ironSource Platform
Registered & Analyzed Data EventsEvery Month
160B
800M
50B
0
100B
150B
200B
Jun 201
5
Jul 201
5
Aug 201
5
Sep 201
5
Oct 201
5
Nov 201
5
Dec 201
5
Jan 201
6
Feb 201
6
Mar 201
6
Apr 201
6
May 201
6
![Page 4: Shimon Tolts, R&D Manager, IronSource - "Data Flow Management"](https://reader036.fdocuments.net/reader036/viewer/2022070516/58713b181a28abf0568b6d21/html5/thumbnails/4.jpg)
We needed a way to manage this data:
Our Business Challenge
ProcessCollect Store
![Page 5: Shimon Tolts, R&D Manager, IronSource - "Data Flow Management"](https://reader036.fdocuments.net/reader036/viewer/2022070516/58713b181a28abf0568b6d21/html5/thumbnails/5.jpg)
Micro-Services Architecture● Everything is a service● Decoupling● Distributed systems
Separate lifecycle● Communication using RESTful /
Queue / Streams
![Page 6: Shimon Tolts, R&D Manager, IronSource - "Data Flow Management"](https://reader036.fdocuments.net/reader036/viewer/2022070516/58713b181a28abf0568b6d21/html5/thumbnails/6.jpg)
Docker● Linux Container● Save provisioning time● Infrastructure as code● Dev-Test-Production - identical
container● Ship easily
![Page 7: Shimon Tolts, R&D Manager, IronSource - "Data Flow Management"](https://reader036.fdocuments.net/reader036/viewer/2022070516/58713b181a28abf0568b6d21/html5/thumbnails/7.jpg)
Cloud infrastructure● Pay as you go - (grow)● SaaS services ● Auto-scaling-groups● DynamoDB● RDS *SQL● Redshift data warehouse
![Page 8: Shimon Tolts, R&D Manager, IronSource - "Data Flow Management"](https://reader036.fdocuments.net/reader036/viewer/2022070516/58713b181a28abf0568b6d21/html5/thumbnails/8.jpg)
Continuous Integration● From commit to production● Jenkins commit hook● Git branching model● AWS dynamic slaves● Unit tests● Docker builds● Updating live environment
![Page 9: Shimon Tolts, R&D Manager, IronSource - "Data Flow Management"](https://reader036.fdocuments.net/reader036/viewer/2022070516/58713b181a28abf0568b6d21/html5/thumbnails/9.jpg)
A lot of beer later….Or, the story of ironSource’s data collection and management solution
![Page 10: Shimon Tolts, R&D Manager, IronSource - "Data Flow Management"](https://reader036.fdocuments.net/reader036/viewer/2022070516/58713b181a28abf0568b6d21/html5/thumbnails/10.jpg)
SDKs
And many more… which one do you need?
![Page 11: Shimon Tolts, R&D Manager, IronSource - "Data Flow Management"](https://reader036.fdocuments.net/reader036/viewer/2022070516/58713b181a28abf0568b6d21/html5/thumbnails/11.jpg)
Collection
● Multi region layer - Latency based
routing
● Low latency from client to Atom servers
● High Availability - AWS regions does
fail!
● Storing raw data + headers upon
receiving
![Page 12: Shimon Tolts, R&D Manager, IronSource - "Data Flow Management"](https://reader036.fdocuments.net/reader036/viewer/2022070516/58713b181a28abf0568b6d21/html5/thumbnails/12.jpg)
Data Enrichment● Enrich data before storing in your Data
Lake and/or Warehouse○ IP to Country○ Currency conversion ○ Decrypt data○ User Agent parsing - OS, Browser, Device...
● Any custom logic you would like! - fully extendible
![Page 13: Shimon Tolts, R&D Manager, IronSource - "Data Flow Management"](https://reader036.fdocuments.net/reader036/viewer/2022070516/58713b181a28abf0568b6d21/html5/thumbnails/13.jpg)
Data Targets● Near real-time data insertion - 1
minute!● Stream data to Google Storage and/or
AWS S3● Smart insertion of data into AWS
Redshift○ Set the amount of parallel copys○ Configure priority on tables
● BigQuery - Streaming data using batch files import (saves 20% cost)
![Page 14: Shimon Tolts, R&D Manager, IronSource - "Data Flow Management"](https://reader036.fdocuments.net/reader036/viewer/2022070516/58713b181a28abf0568b6d21/html5/thumbnails/14.jpg)
David FitcherFrom London, United Kingdom
The User JourneyTouchpoint #1
Customizing anew device
David bought a new LG device.Installed 12 apps overall, 6 were games.
Gender: Male76% Probability
Age: 25-3595% Probability
User Profile: Casual Gamer81% Probability
Touchpoint #2
Using a mobile app
David is now playing a game which uses our SDK.What we already knowProbably a ‘Casual Gamer’What we offerRewarded video ads for casual gaming apps
User Profile: Casual Gamer95% Probability
Subcategory Interest: Simulation GamesLTV Projection: High
One month later…
What we know now
![Page 15: Shimon Tolts, R&D Manager, IronSource - "Data Flow Management"](https://reader036.fdocuments.net/reader036/viewer/2022070516/58713b181a28abf0568b6d21/html5/thumbnails/15.jpg)
Insights
The Data World
Visualization
Data Management
Data Collection
![Page 16: Shimon Tolts, R&D Manager, IronSource - "Data Flow Management"](https://reader036.fdocuments.net/reader036/viewer/2022070516/58713b181a28abf0568b6d21/html5/thumbnails/16.jpg)
![Page 17: Shimon Tolts, R&D Manager, IronSource - "Data Flow Management"](https://reader036.fdocuments.net/reader036/viewer/2022070516/58713b181a28abf0568b6d21/html5/thumbnails/17.jpg)
![Page 18: Shimon Tolts, R&D Manager, IronSource - "Data Flow Management"](https://reader036.fdocuments.net/reader036/viewer/2022070516/58713b181a28abf0568b6d21/html5/thumbnails/18.jpg)
![Page 19: Shimon Tolts, R&D Manager, IronSource - "Data Flow Management"](https://reader036.fdocuments.net/reader036/viewer/2022070516/58713b181a28abf0568b6d21/html5/thumbnails/19.jpg)
Everybody needs a data pipeline
The AWS platform allowed us to build on top of it with an infrastructure that is exactly tailored to our clients’ needs.
Maximum FlexibilityAny data, from any source,
in any format.
Infinite ScalabilityAdapt to your evolving
needs with a pay-as-you-go model.
Own Your DataWe manage the flow,
the data is yours.