CS 195-5: Machine Learning Problem Set 4 1 Support Vector ...
Monitoring of Machine Learning Models Instrumentation, … · 2020-03-22 · Sounds a lot more like...
Transcript of Monitoring of Machine Learning Models Instrumentation, … · 2020-03-22 · Sounds a lot more like...
![Page 1: Monitoring of Machine Learning Models Instrumentation, … · 2020-03-22 · Sounds a lot more like a DevOps problem then a Machine Learning problem to me-e. But really, in general,](https://reader034.fdocuments.net/reader034/viewer/2022042108/5e887f0b52daa55f9d6adf16/html5/thumbnails/1.jpg)
1
Instrumentation, Observability, and Monitoring of Machine Learning Models
![Page 2: Monitoring of Machine Learning Models Instrumentation, … · 2020-03-22 · Sounds a lot more like a DevOps problem then a Machine Learning problem to me-e. But really, in general,](https://reader034.fdocuments.net/reader034/viewer/2022042108/5e887f0b52daa55f9d6adf16/html5/thumbnails/2.jpg)
About Me
● Google Engineer (2007-11)
● Cloudera’s Director of Data Science (2011-15)
● Slack’s Director of Data Engineering (2015-2017)
● Slack Engineer (now)
![Page 3: Monitoring of Machine Learning Models Instrumentation, … · 2020-03-22 · Sounds a lot more like a DevOps problem then a Machine Learning problem to me-e. But really, in general,](https://reader034.fdocuments.net/reader034/viewer/2022042108/5e887f0b52daa55f9d6adf16/html5/thumbnails/3.jpg)
“”
![Page 4: Monitoring of Machine Learning Models Instrumentation, … · 2020-03-22 · Sounds a lot more like a DevOps problem then a Machine Learning problem to me-e. But really, in general,](https://reader034.fdocuments.net/reader034/viewer/2022042108/5e887f0b52daa55f9d6adf16/html5/thumbnails/4.jpg)
“”
![Page 5: Monitoring of Machine Learning Models Instrumentation, … · 2020-03-22 · Sounds a lot more like a DevOps problem then a Machine Learning problem to me-e. But really, in general,](https://reader034.fdocuments.net/reader034/viewer/2022042108/5e887f0b52daa55f9d6adf16/html5/thumbnails/5.jpg)
The Genesis of This Talk
![Page 6: Monitoring of Machine Learning Models Instrumentation, … · 2020-03-22 · Sounds a lot more like a DevOps problem then a Machine Learning problem to me-e. But really, in general,](https://reader034.fdocuments.net/reader034/viewer/2022042108/5e887f0b52daa55f9d6adf16/html5/thumbnails/6.jpg)
Machine Learning In the Wild
![Page 7: Monitoring of Machine Learning Models Instrumentation, … · 2020-03-22 · Sounds a lot more like a DevOps problem then a Machine Learning problem to me-e. But really, in general,](https://reader034.fdocuments.net/reader034/viewer/2022042108/5e887f0b52daa55f9d6adf16/html5/thumbnails/7.jpg)
Data Science Meets DevOps
![Page 8: Monitoring of Machine Learning Models Instrumentation, … · 2020-03-22 · Sounds a lot more like a DevOps problem then a Machine Learning problem to me-e. But really, in general,](https://reader034.fdocuments.net/reader034/viewer/2022042108/5e887f0b52daa55f9d6adf16/html5/thumbnails/8.jpg)
Some History
![Page 9: Monitoring of Machine Learning Models Instrumentation, … · 2020-03-22 · Sounds a lot more like a DevOps problem then a Machine Learning problem to me-e. But really, in general,](https://reader034.fdocuments.net/reader034/viewer/2022042108/5e887f0b52daa55f9d6adf16/html5/thumbnails/9.jpg)
Logs via the ELK Stack
![Page 10: Monitoring of Machine Learning Models Instrumentation, … · 2020-03-22 · Sounds a lot more like a DevOps problem then a Machine Learning problem to me-e. But really, in general,](https://reader034.fdocuments.net/reader034/viewer/2022042108/5e887f0b52daa55f9d6adf16/html5/thumbnails/10.jpg)
Metrics with Prometheus
![Page 11: Monitoring of Machine Learning Models Instrumentation, … · 2020-03-22 · Sounds a lot more like a DevOps problem then a Machine Learning problem to me-e. But really, in general,](https://reader034.fdocuments.net/reader034/viewer/2022042108/5e887f0b52daa55f9d6adf16/html5/thumbnails/11.jpg)
Prometheus Architecture
![Page 12: Monitoring of Machine Learning Models Instrumentation, … · 2020-03-22 · Sounds a lot more like a DevOps problem then a Machine Learning problem to me-e. But really, in general,](https://reader034.fdocuments.net/reader034/viewer/2022042108/5e887f0b52daa55f9d6adf16/html5/thumbnails/12.jpg)
Traces
![Page 13: Monitoring of Machine Learning Models Instrumentation, … · 2020-03-22 · Sounds a lot more like a DevOps problem then a Machine Learning problem to me-e. But really, in general,](https://reader034.fdocuments.net/reader034/viewer/2022042108/5e887f0b52daa55f9d6adf16/html5/thumbnails/13.jpg)
A Word About Cardinality
![Page 14: Monitoring of Machine Learning Models Instrumentation, … · 2020-03-22 · Sounds a lot more like a DevOps problem then a Machine Learning problem to me-e. But really, in general,](https://reader034.fdocuments.net/reader034/viewer/2022042108/5e887f0b52daa55f9d6adf16/html5/thumbnails/14.jpg)
Make Good Decisions By Avoiding Bad Decisions
![Page 15: Monitoring of Machine Learning Models Instrumentation, … · 2020-03-22 · Sounds a lot more like a DevOps problem then a Machine Learning problem to me-e. But really, in general,](https://reader034.fdocuments.net/reader034/viewer/2022042108/5e887f0b52daa55f9d6adf16/html5/thumbnails/15.jpg)
The ML Test Score
![Page 16: Monitoring of Machine Learning Models Instrumentation, … · 2020-03-22 · Sounds a lot more like a DevOps problem then a Machine Learning problem to me-e. But really, in general,](https://reader034.fdocuments.net/reader034/viewer/2022042108/5e887f0b52daa55f9d6adf16/html5/thumbnails/16.jpg)
The Map Is Not The Territory
![Page 17: Monitoring of Machine Learning Models Instrumentation, … · 2020-03-22 · Sounds a lot more like a DevOps problem then a Machine Learning problem to me-e. But really, in general,](https://reader034.fdocuments.net/reader034/viewer/2022042108/5e887f0b52daa55f9d6adf16/html5/thumbnails/17.jpg)
Monitor Model Decay
![Page 18: Monitoring of Machine Learning Models Instrumentation, … · 2020-03-22 · Sounds a lot more like a DevOps problem then a Machine Learning problem to me-e. But really, in general,](https://reader034.fdocuments.net/reader034/viewer/2022042108/5e887f0b52daa55f9d6adf16/html5/thumbnails/18.jpg)
Build Lots of Models
![Page 19: Monitoring of Machine Learning Models Instrumentation, … · 2020-03-22 · Sounds a lot more like a DevOps problem then a Machine Learning problem to me-e. But really, in general,](https://reader034.fdocuments.net/reader034/viewer/2022042108/5e887f0b52daa55f9d6adf16/html5/thumbnails/19.jpg)
Deploy Your Models Like They Are Code*
![Page 20: Monitoring of Machine Learning Models Instrumentation, … · 2020-03-22 · Sounds a lot more like a DevOps problem then a Machine Learning problem to me-e. But really, in general,](https://reader034.fdocuments.net/reader034/viewer/2022042108/5e887f0b52daa55f9d6adf16/html5/thumbnails/20.jpg)
Stand On The Shoulders of Giants
● Ensembles
● Experiments
● Dark Tests
● Canary
● Sanity Checks
![Page 21: Monitoring of Machine Learning Models Instrumentation, … · 2020-03-22 · Sounds a lot more like a DevOps problem then a Machine Learning problem to me-e. But really, in general,](https://reader034.fdocuments.net/reader034/viewer/2022042108/5e887f0b52daa55f9d6adf16/html5/thumbnails/21.jpg)
Tag All The Things
![Page 22: Monitoring of Machine Learning Models Instrumentation, … · 2020-03-22 · Sounds a lot more like a DevOps problem then a Machine Learning problem to me-e. But really, in general,](https://reader034.fdocuments.net/reader034/viewer/2022042108/5e887f0b52daa55f9d6adf16/html5/thumbnails/22.jpg)
Circle of Competence
![Page 23: Monitoring of Machine Learning Models Instrumentation, … · 2020-03-22 · Sounds a lot more like a DevOps problem then a Machine Learning problem to me-e. But really, in general,](https://reader034.fdocuments.net/reader034/viewer/2022042108/5e887f0b52daa55f9d6adf16/html5/thumbnails/23.jpg)
Garbage In...
![Page 24: Monitoring of Machine Learning Models Instrumentation, … · 2020-03-22 · Sounds a lot more like a DevOps problem then a Machine Learning problem to me-e. But really, in general,](https://reader034.fdocuments.net/reader034/viewer/2022042108/5e887f0b52daa55f9d6adf16/html5/thumbnails/24.jpg)
Linking Online and Offline Metrics
![Page 25: Monitoring of Machine Learning Models Instrumentation, … · 2020-03-22 · Sounds a lot more like a DevOps problem then a Machine Learning problem to me-e. But really, in general,](https://reader034.fdocuments.net/reader034/viewer/2022042108/5e887f0b52daa55f9d6adf16/html5/thumbnails/25.jpg)
Handling Cross-Language Feature Engineering
![Page 26: Monitoring of Machine Learning Models Instrumentation, … · 2020-03-22 · Sounds a lot more like a DevOps problem then a Machine Learning problem to me-e. But really, in general,](https://reader034.fdocuments.net/reader034/viewer/2022042108/5e887f0b52daa55f9d6adf16/html5/thumbnails/26.jpg)
Know Your Dependencies
![Page 27: Monitoring of Machine Learning Models Instrumentation, … · 2020-03-22 · Sounds a lot more like a DevOps problem then a Machine Learning problem to me-e. But really, in general,](https://reader034.fdocuments.net/reader034/viewer/2022042108/5e887f0b52daa55f9d6adf16/html5/thumbnails/27.jpg)
Monitoring For Critical Slices
![Page 28: Monitoring of Machine Learning Models Instrumentation, … · 2020-03-22 · Sounds a lot more like a DevOps problem then a Machine Learning problem to me-e. But really, in general,](https://reader034.fdocuments.net/reader034/viewer/2022042108/5e887f0b52daa55f9d6adf16/html5/thumbnails/28.jpg)
Second-Order Thinking
![Page 29: Monitoring of Machine Learning Models Instrumentation, … · 2020-03-22 · Sounds a lot more like a DevOps problem then a Machine Learning problem to me-e. But really, in general,](https://reader034.fdocuments.net/reader034/viewer/2022042108/5e887f0b52daa55f9d6adf16/html5/thumbnails/29.jpg)
On Razors