Scalable Monitoring & Alerting
-
Upload
franklin-angulo -
Category
Technology
-
view
607 -
download
0
description
Transcript of Scalable Monitoring & Alerting
![Page 1: Scalable Monitoring & Alerting](https://reader033.fdocuments.net/reader033/viewer/2022051314/54b70f8a4a795912448b4857/html5/thumbnails/1.jpg)
SCALABLEMONITORING & ALERTING
NEW YORK CITY
![Page 2: Scalable Monitoring & Alerting](https://reader033.fdocuments.net/reader033/viewer/2022051314/54b70f8a4a795912448b4857/html5/thumbnails/2.jpg)
NEW YORK CITY
EXPERIENCE
![Page 3: Scalable Monitoring & Alerting](https://reader033.fdocuments.net/reader033/viewer/2022051314/54b70f8a4a795912448b4857/html5/thumbnails/3.jpg)
THE BEGINNING
NEW YORK CITY
![Page 4: Scalable Monitoring & Alerting](https://reader033.fdocuments.net/reader033/viewer/2022051314/54b70f8a4a795912448b4857/html5/thumbnails/4.jpg)
NEW YORK CITY
DEPLOYMENTS
+ =
![Page 5: Scalable Monitoring & Alerting](https://reader033.fdocuments.net/reader033/viewer/2022051314/54b70f8a4a795912448b4857/html5/thumbnails/5.jpg)
NEW YORK CITY
DDOS ATTACKS
![Page 6: Scalable Monitoring & Alerting](https://reader033.fdocuments.net/reader033/viewer/2022051314/54b70f8a4a795912448b4857/html5/thumbnails/6.jpg)
NEW YORK CITY
THE VISION● Easy publishing of metrics
● Single source of truth
● Democratized access
![Page 7: Scalable Monitoring & Alerting](https://reader033.fdocuments.net/reader033/viewer/2022051314/54b70f8a4a795912448b4857/html5/thumbnails/7.jpg)
NEW YORK CITY
CARBON-AGGREGATOR
CARBON-CACHE
WHISPER
FILESYSTEM (rotating disks)
SEPTEMBER 2013
GRAPHITE WEBAPP
![Page 8: Scalable Monitoring & Alerting](https://reader033.fdocuments.net/reader033/viewer/2022051314/54b70f8a4a795912448b4857/html5/thumbnails/8.jpg)
NEW YORK CITY
16 CARBON-RELAY
10 CARBON-CACHE
WHISPER
FILESYSTEM (rotating)
LOAD BALANCER
WEBAPP
JANUARY 2014
16 CARBON-RELAY
10 CARBON-CACHE
WHISPER
FILESYSTEM (rotating)
WEBAPP
16 CARBON-RELAY
10 CARBON-CACHE
WHISPER
FILESYSTEM (rotating)
WEBAPP
![Page 9: Scalable Monitoring & Alerting](https://reader033.fdocuments.net/reader033/viewer/2022051314/54b70f8a4a795912448b4857/html5/thumbnails/9.jpg)
NEW YORK CITY
CULTURE
![Page 10: Scalable Monitoring & Alerting](https://reader033.fdocuments.net/reader033/viewer/2022051314/54b70f8a4a795912448b4857/html5/thumbnails/10.jpg)
NEW YORK CITY
CULTURE
![Page 11: Scalable Monitoring & Alerting](https://reader033.fdocuments.net/reader033/viewer/2022051314/54b70f8a4a795912448b4857/html5/thumbnails/11.jpg)
SUPER BOWL 2014
NEW YORK CITY
![Page 12: Scalable Monitoring & Alerting](https://reader033.fdocuments.net/reader033/viewer/2022051314/54b70f8a4a795912448b4857/html5/thumbnails/12.jpg)
NEW YORK CITY
![Page 13: Scalable Monitoring & Alerting](https://reader033.fdocuments.net/reader033/viewer/2022051314/54b70f8a4a795912448b4857/html5/thumbnails/13.jpg)
NEW YORK CITY
THE PROBLEMS● Reads didn’t scale
● No usage insight
● Dumb dashboards
![Page 14: Scalable Monitoring & Alerting](https://reader033.fdocuments.net/reader033/viewer/2022051314/54b70f8a4a795912448b4857/html5/thumbnails/14.jpg)
NEW YORK CITY
8 CARBON-RELAY
LOAD BALANCER
APRIL 2014
8 CARBON-RELAY 8 CARBON-RELAY
10 CARBON-CACHE
WHISPER
FILESYSTEM (SSDs)
WEBAPP
![Page 15: Scalable Monitoring & Alerting](https://reader033.fdocuments.net/reader033/viewer/2022051314/54b70f8a4a795912448b4857/html5/thumbnails/15.jpg)
NEW YORK CITY
8 CARBON-RELAY
LOAD BALANCER
MAY 2014
8 CARBON-RELAY 8 CARBON-RELAY
10 CARBON-CACHE
WHISPER
FILESYSTEM (SSDs)
GRAPHITE API
![Page 16: Scalable Monitoring & Alerting](https://reader033.fdocuments.net/reader033/viewer/2022051314/54b70f8a4a795912448b4857/html5/thumbnails/16.jpg)
NEW YORK CITY
8 CARBON-RELAY
LOAD BALANCER
JUNE 2014
8 CARBON-RELAY
8 CARBON-RELAY
10 CARBON-CACHE
WHISPER
FILESYSTEM (SSDs)
GRAPHITE API
10 CARBON-AGGREGATOR 10 CARBON-AGGREGATOR
10 CARBON-AGGREGATOR
![Page 17: Scalable Monitoring & Alerting](https://reader033.fdocuments.net/reader033/viewer/2022051314/54b70f8a4a795912448b4857/html5/thumbnails/17.jpg)
NEW YORK CITY
USAGE INFORMATION
+
=
GRAPHITE API + +
![Page 18: Scalable Monitoring & Alerting](https://reader033.fdocuments.net/reader033/viewer/2022051314/54b70f8a4a795912448b4857/html5/thumbnails/18.jpg)
NEW YORK CITY
![Page 19: Scalable Monitoring & Alerting](https://reader033.fdocuments.net/reader033/viewer/2022051314/54b70f8a4a795912448b4857/html5/thumbnails/19.jpg)
NEW YORK CITYGRAFANA
![Page 20: Scalable Monitoring & Alerting](https://reader033.fdocuments.net/reader033/viewer/2022051314/54b70f8a4a795912448b4857/html5/thumbnails/20.jpg)
ALERTING
NEW YORK CITY
![Page 21: Scalable Monitoring & Alerting](https://reader033.fdocuments.net/reader033/viewer/2022051314/54b70f8a4a795912448b4857/html5/thumbnails/21.jpg)
Core InfrastructureTHRESHOLDS
CRITICAL THRESHOLD
WARNING THRESHOLD
![Page 22: Scalable Monitoring & Alerting](https://reader033.fdocuments.net/reader033/viewer/2022051314/54b70f8a4a795912448b4857/html5/thumbnails/22.jpg)
Core InfrastructureALERTING MATH
![Page 23: Scalable Monitoring & Alerting](https://reader033.fdocuments.net/reader033/viewer/2022051314/54b70f8a4a795912448b4857/html5/thumbnails/23.jpg)
Core InfrastructureALERTING MATH
SkipRead30 5
Alarm9
![Page 24: Scalable Monitoring & Alerting](https://reader033.fdocuments.net/reader033/viewer/2022051314/54b70f8a4a795912448b4857/html5/thumbnails/24.jpg)
Core InfrastructureALERTING MATH
SkipRead30 5
Frequency10 sec
300 secs 50 secs
Alarm9
90 secs
![Page 25: Scalable Monitoring & Alerting](https://reader033.fdocuments.net/reader033/viewer/2022051314/54b70f8a4a795912448b4857/html5/thumbnails/25.jpg)
Core InfrastructureNAGIOS INTEGRATION
![Page 26: Scalable Monitoring & Alerting](https://reader033.fdocuments.net/reader033/viewer/2022051314/54b70f8a4a795912448b4857/html5/thumbnails/26.jpg)
NEW YORK CITY
NEXT STEPS● Replication
● Scale even more
● More robust alerting logic
![Page 27: Scalable Monitoring & Alerting](https://reader033.fdocuments.net/reader033/viewer/2022051314/54b70f8a4a795912448b4857/html5/thumbnails/27.jpg)
NEW YORK CITY
8 CARBON-RELAY
LOAD BALANCER
REPLICATION
8 CARBON-RELAY
8 CARBON-RELAY
10 CARBON-CACHE
WHISPER
FILESYSTEM (SSDs)
GRAPHITE API
10 CARBON-AGGREGATOR 10 CARBON-AGGREGATOR
10 CARBON-AGGREGATOR
![Page 28: Scalable Monitoring & Alerting](https://reader033.fdocuments.net/reader033/viewer/2022051314/54b70f8a4a795912448b4857/html5/thumbnails/28.jpg)
Core InfrastructureALERTING LOGIC
= Google Forms Throttling
![Page 29: Scalable Monitoring & Alerting](https://reader033.fdocuments.net/reader033/viewer/2022051314/54b70f8a4a795912448b4857/html5/thumbnails/29.jpg)
Core InfrastructureALERTING LOGIC
Trial subscription signups
Internal subscription API calls
Database host offline
User
Application
System
![Page 30: Scalable Monitoring & Alerting](https://reader033.fdocuments.net/reader033/viewer/2022051314/54b70f8a4a795912448b4857/html5/thumbnails/30.jpg)
NEW YORK CITYLEARN MORE
![Page 31: Scalable Monitoring & Alerting](https://reader033.fdocuments.net/reader033/viewer/2022051314/54b70f8a4a795912448b4857/html5/thumbnails/31.jpg)
QUESTIONS
NEW YORK CITY