Resources and Guidelines Building Data Products€¦ · Building Data Products Resources and...
Transcript of Resources and Guidelines Building Data Products€¦ · Building Data Products Resources and...
![Page 1: Resources and Guidelines Building Data Products€¦ · Building Data Products Resources and Guidelines Bob Raman, Engineering Manager - Data . Today’s agenda 2. Answer Bot Data](https://reader035.fdocuments.net/reader035/viewer/2022070511/5fe626e7b6b86a728c03a650/html5/thumbnails/1.jpg)
Learnings from Building Data ProductsResources and Guidelines
Bob Raman, Engineering Manager - Data
![Page 2: Resources and Guidelines Building Data Products€¦ · Building Data Products Resources and Guidelines Bob Raman, Engineering Manager - Data . Today’s agenda 2. Answer Bot Data](https://reader035.fdocuments.net/reader035/viewer/2022070511/5fe626e7b6b86a728c03a650/html5/thumbnails/2.jpg)
Today’s agenda
2. Answer Bot
Data Product Team
1. Customer Satisfaction (CSAT)
3. Team structure, Delivery, Scalability
4. Our next evolution in building Data Products
5. Takeaways
![Page 3: Resources and Guidelines Building Data Products€¦ · Building Data Products Resources and Guidelines Bob Raman, Engineering Manager - Data . Today’s agenda 2. Answer Bot Data](https://reader035.fdocuments.net/reader035/viewer/2022070511/5fe626e7b6b86a728c03a650/html5/thumbnails/3.jpg)
Customer engagement
Relationship management
![Page 4: Resources and Guidelines Building Data Products€¦ · Building Data Products Resources and Guidelines Bob Raman, Engineering Manager - Data . Today’s agenda 2. Answer Bot Data](https://reader035.fdocuments.net/reader035/viewer/2022070511/5fe626e7b6b86a728c03a650/html5/thumbnails/4.jpg)
Zendesk
![Page 5: Resources and Guidelines Building Data Products€¦ · Building Data Products Resources and Guidelines Bob Raman, Engineering Manager - Data . Today’s agenda 2. Answer Bot Data](https://reader035.fdocuments.net/reader035/viewer/2022070511/5fe626e7b6b86a728c03a650/html5/thumbnails/5.jpg)
Data Product Team
● Build machine learning based products to solve business problems
● Team of 20+ people○ Data Eng - full stack
![Page 6: Resources and Guidelines Building Data Products€¦ · Building Data Products Resources and Guidelines Bob Raman, Engineering Manager - Data . Today’s agenda 2. Answer Bot Data](https://reader035.fdocuments.net/reader035/viewer/2022070511/5fe626e7b6b86a728c03a650/html5/thumbnails/6.jpg)
THE
TEA
MC
irca
6/2
017
![Page 7: Resources and Guidelines Building Data Products€¦ · Building Data Products Resources and Guidelines Bob Raman, Engineering Manager - Data . Today’s agenda 2. Answer Bot Data](https://reader035.fdocuments.net/reader035/viewer/2022070511/5fe626e7b6b86a728c03a650/html5/thumbnails/7.jpg)
Satisfaction Prediction (CSAT)Identify end users at risk of having a bad outcome before it happens
![Page 8: Resources and Guidelines Building Data Products€¦ · Building Data Products Resources and Guidelines Bob Raman, Engineering Manager - Data . Today’s agenda 2. Answer Bot Data](https://reader035.fdocuments.net/reader035/viewer/2022070511/5fe626e7b6b86a728c03a650/html5/thumbnails/8.jpg)
CSAT in Action
![Page 9: Resources and Guidelines Building Data Products€¦ · Building Data Products Resources and Guidelines Bob Raman, Engineering Manager - Data . Today’s agenda 2. Answer Bot Data](https://reader035.fdocuments.net/reader035/viewer/2022070511/5fe626e7b6b86a728c03a650/html5/thumbnails/9.jpg)
CSAT Thresholding
![Page 10: Resources and Guidelines Building Data Products€¦ · Building Data Products Resources and Guidelines Bob Raman, Engineering Manager - Data . Today’s agenda 2. Answer Bot Data](https://reader035.fdocuments.net/reader035/viewer/2022070511/5fe626e7b6b86a728c03a650/html5/thumbnails/10.jpg)
Background - Classical Machine Learning Model
![Page 11: Resources and Guidelines Building Data Products€¦ · Building Data Products Resources and Guidelines Bob Raman, Engineering Manager - Data . Today’s agenda 2. Answer Bot Data](https://reader035.fdocuments.net/reader035/viewer/2022070511/5fe626e7b6b86a728c03a650/html5/thumbnails/11.jpg)
Background - Model Build Stack
Map-Reduce - Cascalog (Clojure)
![Page 12: Resources and Guidelines Building Data Products€¦ · Building Data Products Resources and Guidelines Bob Raman, Engineering Manager - Data . Today’s agenda 2. Answer Bot Data](https://reader035.fdocuments.net/reader035/viewer/2022070511/5fe626e7b6b86a728c03a650/html5/thumbnails/12.jpg)
Background - Model Serving Stack (Change Data Capture)
Serving machine learning models on AWS
![Page 13: Resources and Guidelines Building Data Products€¦ · Building Data Products Resources and Guidelines Bob Raman, Engineering Manager - Data . Today’s agenda 2. Answer Bot Data](https://reader035.fdocuments.net/reader035/viewer/2022070511/5fe626e7b6b86a728c03a650/html5/thumbnails/13.jpg)
How we worked?
Data Scientists and Data Engineers
● Separate DS and DE teams○ 3 Data Scientists (DS)○ 5 Data Engineers (DE)○ PM, QA, Shared UX and Ops
● PM worked with DS and DE teams
![Page 14: Resources and Guidelines Building Data Products€¦ · Building Data Products Resources and Guidelines Bob Raman, Engineering Manager - Data . Today’s agenda 2. Answer Bot Data](https://reader035.fdocuments.net/reader035/viewer/2022070511/5fe626e7b6b86a728c03a650/html5/thumbnails/14.jpg)
Delivery - Streams
![Page 15: Resources and Guidelines Building Data Products€¦ · Building Data Products Resources and Guidelines Bob Raman, Engineering Manager - Data . Today’s agenda 2. Answer Bot Data](https://reader035.fdocuments.net/reader035/viewer/2022070511/5fe626e7b6b86a728c03a650/html5/thumbnails/15.jpg)
Delivery - Scalability
● Model serving ○ “Dark Launch” approach
● Model building ○ Use Map-reduce - 3000 models/week■ Shell out in “reducer”
![Page 16: Resources and Guidelines Building Data Products€¦ · Building Data Products Resources and Guidelines Bob Raman, Engineering Manager - Data . Today’s agenda 2. Answer Bot Data](https://reader035.fdocuments.net/reader035/viewer/2022070511/5fe626e7b6b86a728c03a650/html5/thumbnails/16.jpg)
Learnings #1 - Technology
● Data Scientists deploy python to prod○ Python is fast enough○ Virtual environment - Virtualenv○ Scaling done by Data Engineers
● De-normalised intermediate datasets
![Page 17: Resources and Guidelines Building Data Products€¦ · Building Data Products Resources and Guidelines Bob Raman, Engineering Manager - Data . Today’s agenda 2. Answer Bot Data](https://reader035.fdocuments.net/reader035/viewer/2022070511/5fe626e7b6b86a728c03a650/html5/thumbnails/17.jpg)
Learnings #2 - UX Issues
● Exposing CSAT score ○ Fragile customer business rules■ Limits improvement to algorithm○ Gradated scale may have been better
![Page 18: Resources and Guidelines Building Data Products€¦ · Building Data Products Resources and Guidelines Bob Raman, Engineering Manager - Data . Today’s agenda 2. Answer Bot Data](https://reader035.fdocuments.net/reader035/viewer/2022070511/5fe626e7b6b86a728c03a650/html5/thumbnails/18.jpg)
Model-UX conundrum
![Page 19: Resources and Guidelines Building Data Products€¦ · Building Data Products Resources and Guidelines Bob Raman, Engineering Manager - Data . Today’s agenda 2. Answer Bot Data](https://reader035.fdocuments.net/reader035/viewer/2022070511/5fe626e7b6b86a728c03a650/html5/thumbnails/19.jpg)
Learnings #4 - Project Delivery
● Need roadmap alignment across teams
● Infra compliance increased scope
![Page 20: Resources and Guidelines Building Data Products€¦ · Building Data Products Resources and Guidelines Bob Raman, Engineering Manager - Data . Today’s agenda 2. Answer Bot Data](https://reader035.fdocuments.net/reader035/viewer/2022070511/5fe626e7b6b86a728c03a650/html5/thumbnails/20.jpg)
Answer BotAutomatically answer end-user problems
![Page 21: Resources and Guidelines Building Data Products€¦ · Building Data Products Resources and Guidelines Bob Raman, Engineering Manager - Data . Today’s agenda 2. Answer Bot Data](https://reader035.fdocuments.net/reader035/viewer/2022070511/5fe626e7b6b86a728c03a650/html5/thumbnails/21.jpg)
Answer Bot - Customer emails in a question
![Page 22: Resources and Guidelines Building Data Products€¦ · Building Data Products Resources and Guidelines Bob Raman, Engineering Manager - Data . Today’s agenda 2. Answer Bot Data](https://reader035.fdocuments.net/reader035/viewer/2022070511/5fe626e7b6b86a728c03a650/html5/thumbnails/22.jpg)
Answer Bot - Suggests most relevant articles
![Page 23: Resources and Guidelines Building Data Products€¦ · Building Data Products Resources and Guidelines Bob Raman, Engineering Manager - Data . Today’s agenda 2. Answer Bot Data](https://reader035.fdocuments.net/reader035/viewer/2022070511/5fe626e7b6b86a728c03a650/html5/thumbnails/23.jpg)
How we worked?
● Lean startup ○ “Only way to win is learn faster than
anyone else.” [Eric Ries]
● +1 DS, +2 DE, Dedicated UX
![Page 24: Resources and Guidelines Building Data Products€¦ · Building Data Products Resources and Guidelines Bob Raman, Engineering Manager - Data . Today’s agenda 2. Answer Bot Data](https://reader035.fdocuments.net/reader035/viewer/2022070511/5fe626e7b6b86a728c03a650/html5/thumbnails/24.jpg)
MVP Hypothesis
Examples
● End users will interact with links in email
● Wrong answer does not negatively affect customer experience
● ...
![Page 25: Resources and Guidelines Building Data Products€¦ · Building Data Products Resources and Guidelines Bob Raman, Engineering Manager - Data . Today’s agenda 2. Answer Bot Data](https://reader035.fdocuments.net/reader035/viewer/2022070511/5fe626e7b6b86a728c03a650/html5/thumbnails/25.jpg)
MVP Measurements
Examples
● Surveys○ We have large number of customers
● Cohort metrics - “Activate and left on”○ New○ Oldie○ No-hoper
![Page 26: Resources and Guidelines Building Data Products€¦ · Building Data Products Resources and Guidelines Bob Raman, Engineering Manager - Data . Today’s agenda 2. Answer Bot Data](https://reader035.fdocuments.net/reader035/viewer/2022070511/5fe626e7b6b86a728c03a650/html5/thumbnails/26.jpg)
Background - Deep Learning Model
● Deep learning neural network model○ Measure similarity
between articles and tickets
○ Google TensorFlow (TF)
![Page 27: Resources and Guidelines Building Data Products€¦ · Building Data Products Resources and Guidelines Bob Raman, Engineering Manager - Data . Today’s agenda 2. Answer Bot Data](https://reader035.fdocuments.net/reader035/viewer/2022070511/5fe626e7b6b86a728c03a650/html5/thumbnails/27.jpg)
Background - Model Serving Stack (Change Data Capture)
Serving and updating machine learning model in real time on AWS
![Page 28: Resources and Guidelines Building Data Products€¦ · Building Data Products Resources and Guidelines Bob Raman, Engineering Manager - Data . Today’s agenda 2. Answer Bot Data](https://reader035.fdocuments.net/reader035/viewer/2022070511/5fe626e7b6b86a728c03a650/html5/thumbnails/28.jpg)
Delivery - Streams
![Page 29: Resources and Guidelines Building Data Products€¦ · Building Data Products Resources and Guidelines Bob Raman, Engineering Manager - Data . Today’s agenda 2. Answer Bot Data](https://reader035.fdocuments.net/reader035/viewer/2022070511/5fe626e7b6b86a728c03a650/html5/thumbnails/29.jpg)
Delivery - Key Items
● Single DL model for all customers ● Real time update of models● Self-service● Roadmap alignment across teams○ UX in AU○ 7 dev teams from EU, US, AU
![Page 30: Resources and Guidelines Building Data Products€¦ · Building Data Products Resources and Guidelines Bob Raman, Engineering Manager - Data . Today’s agenda 2. Answer Bot Data](https://reader035.fdocuments.net/reader035/viewer/2022070511/5fe626e7b6b86a728c03a650/html5/thumbnails/30.jpg)
Delivery - Scalability
● Real-time update of models○ Capacity planning - articles ’s/min○ TensorFlow Model■ CPU’s - prediction■ GPU’s - updating model
![Page 31: Resources and Guidelines Building Data Products€¦ · Building Data Products Resources and Guidelines Bob Raman, Engineering Manager - Data . Today’s agenda 2. Answer Bot Data](https://reader035.fdocuments.net/reader035/viewer/2022070511/5fe626e7b6b86a728c03a650/html5/thumbnails/31.jpg)
Learnings #1 - MVP’s
● Dogfooding Zendesk App○ Fast to build: 2-4 weeks
● Do not build MVP’s that negatively affect primary customer workflow
![Page 32: Resources and Guidelines Building Data Products€¦ · Building Data Products Resources and Guidelines Bob Raman, Engineering Manager - Data . Today’s agenda 2. Answer Bot Data](https://reader035.fdocuments.net/reader035/viewer/2022070511/5fe626e7b6b86a728c03a650/html5/thumbnails/32.jpg)
Learnings #2 - Moving faster with ML model development
● “Gold” dataset for validation
● A/B testing of models
● Seperate model and UX tests
![Page 33: Resources and Guidelines Building Data Products€¦ · Building Data Products Resources and Guidelines Bob Raman, Engineering Manager - Data . Today’s agenda 2. Answer Bot Data](https://reader035.fdocuments.net/reader035/viewer/2022070511/5fe626e7b6b86a728c03a650/html5/thumbnails/33.jpg)
Challenges - Model-UX conundrum
![Page 34: Resources and Guidelines Building Data Products€¦ · Building Data Products Resources and Guidelines Bob Raman, Engineering Manager - Data . Today’s agenda 2. Answer Bot Data](https://reader035.fdocuments.net/reader035/viewer/2022070511/5fe626e7b6b86a728c03a650/html5/thumbnails/34.jpg)
Learnings #3 - Lean Startup
● Build metrics into the product
● Use “Goal” driven iterations vs points
![Page 35: Resources and Guidelines Building Data Products€¦ · Building Data Products Resources and Guidelines Bob Raman, Engineering Manager - Data . Today’s agenda 2. Answer Bot Data](https://reader035.fdocuments.net/reader035/viewer/2022070511/5fe626e7b6b86a728c03a650/html5/thumbnails/35.jpg)
Example: Metrics picked up probs with UX
![Page 36: Resources and Guidelines Building Data Products€¦ · Building Data Products Resources and Guidelines Bob Raman, Engineering Manager - Data . Today’s agenda 2. Answer Bot Data](https://reader035.fdocuments.net/reader035/viewer/2022070511/5fe626e7b6b86a728c03a650/html5/thumbnails/36.jpg)
Metrics - Datadog
![Page 37: Resources and Guidelines Building Data Products€¦ · Building Data Products Resources and Guidelines Bob Raman, Engineering Manager - Data . Today’s agenda 2. Answer Bot Data](https://reader035.fdocuments.net/reader035/viewer/2022070511/5fe626e7b6b86a728c03a650/html5/thumbnails/37.jpg)
Monthly Recurring Revenue Targets Q3 2017
![Page 38: Resources and Guidelines Building Data Products€¦ · Building Data Products Resources and Guidelines Bob Raman, Engineering Manager - Data . Today’s agenda 2. Answer Bot Data](https://reader035.fdocuments.net/reader035/viewer/2022070511/5fe626e7b6b86a728c03a650/html5/thumbnails/38.jpg)
![Page 39: Resources and Guidelines Building Data Products€¦ · Building Data Products Resources and Guidelines Bob Raman, Engineering Manager - Data . Today’s agenda 2. Answer Bot Data](https://reader035.fdocuments.net/reader035/viewer/2022070511/5fe626e7b6b86a728c03a650/html5/thumbnails/39.jpg)
Can we do better with teams?
● No Silos● Blur line between DS and DE● DS and DE solve problems together● Build kickass team by building trust
![Page 40: Resources and Guidelines Building Data Products€¦ · Building Data Products Resources and Guidelines Bob Raman, Engineering Manager - Data . Today’s agenda 2. Answer Bot Data](https://reader035.fdocuments.net/reader035/viewer/2022070511/5fe626e7b6b86a728c03a650/html5/thumbnails/40.jpg)
Product based teams
● 2 Pizza size cross-functional squads ○ Embedded Scientists, Ops, UX○ PM per squad● End-end ownership○ From Research to Delivery● Single backlog● Co-located
![Page 41: Resources and Guidelines Building Data Products€¦ · Building Data Products Resources and Guidelines Bob Raman, Engineering Manager - Data . Today’s agenda 2. Answer Bot Data](https://reader035.fdocuments.net/reader035/viewer/2022070511/5fe626e7b6b86a728c03a650/html5/thumbnails/41.jpg)
Team setup
● Teams in Production○ 3:1 DE/DS ratio○ Delivery lead - Engineering
● Teams in research○ 2:1 DE/DS ratio ○ Delivery lead - Data Science
![Page 42: Resources and Guidelines Building Data Products€¦ · Building Data Products Resources and Guidelines Bob Raman, Engineering Manager - Data . Today’s agenda 2. Answer Bot Data](https://reader035.fdocuments.net/reader035/viewer/2022070511/5fe626e7b6b86a728c03a650/html5/thumbnails/42.jpg)
Team setup
● Teams in Production○ 3:1 DE/DS ratio○ Delivery lead - Engineering
● Teams in research○ 2:1 DE/DS ratio ○ Delivery lead - Data Science
![Page 43: Resources and Guidelines Building Data Products€¦ · Building Data Products Resources and Guidelines Bob Raman, Engineering Manager - Data . Today’s agenda 2. Answer Bot Data](https://reader035.fdocuments.net/reader035/viewer/2022070511/5fe626e7b6b86a728c03a650/html5/thumbnails/43.jpg)
Next Platform - in research
• GCP • DS’s
• Datalab• Dataprep
• DE’s• Dataflow, BigQuery, BigTable• GKE - K8• TPU’s (we have invested in TF)
![Page 44: Resources and Guidelines Building Data Products€¦ · Building Data Products Resources and Guidelines Bob Raman, Engineering Manager - Data . Today’s agenda 2. Answer Bot Data](https://reader035.fdocuments.net/reader035/viewer/2022070511/5fe626e7b6b86a728c03a650/html5/thumbnails/44.jpg)
Takeaway’s
![Page 45: Resources and Guidelines Building Data Products€¦ · Building Data Products Resources and Guidelines Bob Raman, Engineering Manager - Data . Today’s agenda 2. Answer Bot Data](https://reader035.fdocuments.net/reader035/viewer/2022070511/5fe626e7b6b86a728c03a650/html5/thumbnails/45.jpg)
Key Takeaway’s
● Be clear with your hypothesis○ Are you validating the model?○ Or are you testing UX?
● Build a culture of data driven decision making
![Page 46: Resources and Guidelines Building Data Products€¦ · Building Data Products Resources and Guidelines Bob Raman, Engineering Manager - Data . Today’s agenda 2. Answer Bot Data](https://reader035.fdocuments.net/reader035/viewer/2022070511/5fe626e7b6b86a728c03a650/html5/thumbnails/46.jpg)
Key Takeaway’s
● “Gold” dataset can help you go faster
● DS deliver code to production
● “Goal” driven iterations in research
![Page 47: Resources and Guidelines Building Data Products€¦ · Building Data Products Resources and Guidelines Bob Raman, Engineering Manager - Data . Today’s agenda 2. Answer Bot Data](https://reader035.fdocuments.net/reader035/viewer/2022070511/5fe626e7b6b86a728c03a650/html5/thumbnails/47.jpg)
Acknowledgements
Thanks to the awesome Data Product team
● Ai-Lien Trong-Cong● Arvind Kunday● Anh Dinh● Arwen Griffioen● Balaji Sekar● Chris Hausler● Chris Holman● Christine Lin● Damen Turnbull● Derrick Cheng● Ed Savage● Eric Pak● Jason Maynard
● Jeff Theobald● Jeremy Kirkwood● Mike Mortimer● Paul Gradie● Patsy Lai● Pei Shi Yong● Sean Caffery● Shiyu Zhu● Soon-ee Cheah● Tan Le● Wai Chee Yau
![Page 48: Resources and Guidelines Building Data Products€¦ · Building Data Products Resources and Guidelines Bob Raman, Engineering Manager - Data . Today’s agenda 2. Answer Bot Data](https://reader035.fdocuments.net/reader035/viewer/2022070511/5fe626e7b6b86a728c03a650/html5/thumbnails/48.jpg)
TM and © 2017 Zendesk Inc. All rights reserved.
Questions?