Mesos%@Bloomberg · InfluxDB Grafana Monitoring%Applications. Mesos Slave Log Shipper App1 App2...
Transcript of Mesos%@Bloomberg · InfluxDB Grafana Monitoring%Applications. Mesos Slave Log Shipper App1 App2...
![Page 1: Mesos%@Bloomberg · InfluxDB Grafana Monitoring%Applications. Mesos Slave Log Shipper App1 App2 App3 StatsD CollectD Kafka Master Master Master ELK InfluxDB Grafana Monitoring%Applications.](https://reader035.fdocuments.net/reader035/viewer/2022062919/5ee0f063ad6a402d666bffbc/html5/thumbnails/1.jpg)
Mesos @ BloombergMesosCon 2015
Skand S GuptaBloomberg LP
![Page 2: Mesos%@Bloomberg · InfluxDB Grafana Monitoring%Applications. Mesos Slave Log Shipper App1 App2 App3 StatsD CollectD Kafka Master Master Master ELK InfluxDB Grafana Monitoring%Applications.](https://reader035.fdocuments.net/reader035/viewer/2022062919/5ee0f063ad6a402d666bffbc/html5/thumbnails/2.jpg)
Bloomberg
• Bloomberg technology helps drive the world’s financial markets – We run one of the world’s largest private network with over 20,000 routers across our network
– We developed “cloud computing” and deployed “software as a service” well ahead of the general marketplace
– Our technology, has brought transparency to the global financial markets
• Bloomberg technologists – More than 3,000 software developers and designers located around the world (London, NYC, SF)
– BloombergLabs.com (@BloombergLabs) is our platform for dialogue between our experts and the broader tech community
• Our clients – Over 320,000 subscribers – Primarily financial professionals including investment bankers, CFOs, investor relations, hedge funds managers, foreign exchange, etc.
Copyright 2015 Bloomberg L.P.
![Page 3: Mesos%@Bloomberg · InfluxDB Grafana Monitoring%Applications. Mesos Slave Log Shipper App1 App2 App3 StatsD CollectD Kafka Master Master Master ELK InfluxDB Grafana Monitoring%Applications.](https://reader035.fdocuments.net/reader035/viewer/2022062919/5ee0f063ad6a402d666bffbc/html5/thumbnails/3.jpg)
Banks Reported Flawed Interest Data for LIBOR
(WSJ)
2008
![Page 4: Mesos%@Bloomberg · InfluxDB Grafana Monitoring%Applications. Mesos Slave Log Shipper App1 App2 App3 StatsD CollectD Kafka Master Master Master ELK InfluxDB Grafana Monitoring%Applications.](https://reader035.fdocuments.net/reader035/viewer/2022062919/5ee0f063ad6a402d666bffbc/html5/thumbnails/4.jpg)
LIBOR
• Impacts consumer lending such as mortgages, student loans, etc. (LIBOR + ~3%)
• $450 Trillion worth of financial deals are dependent on it • Measure of trust in financial system
![Page 5: Mesos%@Bloomberg · InfluxDB Grafana Monitoring%Applications. Mesos Slave Log Shipper App1 App2 App3 StatsD CollectD Kafka Master Master Master ELK InfluxDB Grafana Monitoring%Applications.](https://reader035.fdocuments.net/reader035/viewer/2022062919/5ee0f063ad6a402d666bffbc/html5/thumbnails/5.jpg)
Banks Reported Flawed Interest Data for LIBOR
(WSJ)
2008
CFTC Orders Barclays to Pay
$200 Million Fine (CFTC)
2012
![Page 6: Mesos%@Bloomberg · InfluxDB Grafana Monitoring%Applications. Mesos Slave Log Shipper App1 App2 App3 StatsD CollectD Kafka Master Master Master ELK InfluxDB Grafana Monitoring%Applications.](https://reader035.fdocuments.net/reader035/viewer/2022062919/5ee0f063ad6a402d666bffbc/html5/thumbnails/6.jpg)
Banks Reported Flawed Interest Data for LIBOR
(WSJ)
2008
CFTC Orders Barclays to Pay
$200 Million Fine (CFTC)
2012
Deutsche Bank Pays $2.5 Billion
to Settle Rate-Rigging Case
(NYT)
2015
![Page 7: Mesos%@Bloomberg · InfluxDB Grafana Monitoring%Applications. Mesos Slave Log Shipper App1 App2 App3 StatsD CollectD Kafka Master Master Master ELK InfluxDB Grafana Monitoring%Applications.](https://reader035.fdocuments.net/reader035/viewer/2022062919/5ee0f063ad6a402d666bffbc/html5/thumbnails/7.jpg)
Banks Reported Flawed Interest Data for LIBOR
(WSJ)
2008
CFTC Orders Barclays to Pay
$200 Million Fine (CFTC)
2012
Deutsche Bank Pays $2.5 Billion
to Settle Rate-Rigging Case
(NYT)
2015
Fines Totaling $6 Billion!
![Page 8: Mesos%@Bloomberg · InfluxDB Grafana Monitoring%Applications. Mesos Slave Log Shipper App1 App2 App3 StatsD CollectD Kafka Master Master Master ELK InfluxDB Grafana Monitoring%Applications.](https://reader035.fdocuments.net/reader035/viewer/2022062919/5ee0f063ad6a402d666bffbc/html5/thumbnails/8.jpg)
Source: Commodity Futures Trading Commission
“The Cartel”
![Page 9: Mesos%@Bloomberg · InfluxDB Grafana Monitoring%Applications. Mesos Slave Log Shipper App1 App2 App3 StatsD CollectD Kafka Master Master Master ELK InfluxDB Grafana Monitoring%Applications.](https://reader035.fdocuments.net/reader035/viewer/2022062919/5ee0f063ad6a402d666bffbc/html5/thumbnails/9.jpg)
Compliance Platform and Processing Pipeline
Chat
Reference Data
Trade Data
Customer Data
Product Data
Market Data
Counterparty
Social Media Voice
Human-‐ and Machine-‐generated Data
Surveillance Pipeline
Communication Data
Transactional Data
User Data
Case Management
Compliance Platform
Compliance Storage
Compliance Officers
Search, Review, Analyze
![Page 10: Mesos%@Bloomberg · InfluxDB Grafana Monitoring%Applications. Mesos Slave Log Shipper App1 App2 App3 StatsD CollectD Kafka Master Master Master ELK InfluxDB Grafana Monitoring%Applications.](https://reader035.fdocuments.net/reader035/viewer/2022062919/5ee0f063ad6a402d666bffbc/html5/thumbnails/10.jpg)
Work Loads – Complex Event Processing
Policies
•Real Time Low Latency Processing •CPU bound
![Page 11: Mesos%@Bloomberg · InfluxDB Grafana Monitoring%Applications. Mesos Slave Log Shipper App1 App2 App3 StatsD CollectD Kafka Master Master Master ELK InfluxDB Grafana Monitoring%Applications.](https://reader035.fdocuments.net/reader035/viewer/2022062919/5ee0f063ad6a402d666bffbc/html5/thumbnails/11.jpg)
Work Loads – Text Searches
Text Index
Tokens
•Distributed •Memory bound
![Page 12: Mesos%@Bloomberg · InfluxDB Grafana Monitoring%Applications. Mesos Slave Log Shipper App1 App2 App3 StatsD CollectD Kafka Master Master Master ELK InfluxDB Grafana Monitoring%Applications.](https://reader035.fdocuments.net/reader035/viewer/2022062919/5ee0f063ad6a402d666bffbc/html5/thumbnails/12.jpg)
Work Loads – Analytics & Reporting
•Ad Hoc •Distributed •Reporting: I/O bound •Analytics: CPU bound
![Page 13: Mesos%@Bloomberg · InfluxDB Grafana Monitoring%Applications. Mesos Slave Log Shipper App1 App2 App3 StatsD CollectD Kafka Master Master Master ELK InfluxDB Grafana Monitoring%Applications.](https://reader035.fdocuments.net/reader035/viewer/2022062919/5ee0f063ad6a402d666bffbc/html5/thumbnails/13.jpg)
Work LoadsPolicies
• Real Time Low Latency Processing • CPU bound
Text Index
Tokens
• Distributed •Memory bound
• Ad Hoc • Distributed • Reporting: I/O bound • Analytics: CPU bound
![Page 14: Mesos%@Bloomberg · InfluxDB Grafana Monitoring%Applications. Mesos Slave Log Shipper App1 App2 App3 StatsD CollectD Kafka Master Master Master ELK InfluxDB Grafana Monitoring%Applications.](https://reader035.fdocuments.net/reader035/viewer/2022062919/5ee0f063ad6a402d666bffbc/html5/thumbnails/14.jpg)
Complex Event Processing
Before Mesos
Text Search
Reporting & Analytics
Lights up during market hours
Idle, till its not!
Can’t get enough resources
![Page 15: Mesos%@Bloomberg · InfluxDB Grafana Monitoring%Applications. Mesos Slave Log Shipper App1 App2 App3 StatsD CollectD Kafka Master Master Master ELK InfluxDB Grafana Monitoring%Applications.](https://reader035.fdocuments.net/reader035/viewer/2022062919/5ee0f063ad6a402d666bffbc/html5/thumbnails/15.jpg)
Before Mesos
Complex Event Processing Text Search
Reporting & Analytics
• Time consuming to re-‐size a static cluster • Wasted resources • Operational overhead
![Page 16: Mesos%@Bloomberg · InfluxDB Grafana Monitoring%Applications. Mesos Slave Log Shipper App1 App2 App3 StatsD CollectD Kafka Master Master Master ELK InfluxDB Grafana Monitoring%Applications.](https://reader035.fdocuments.net/reader035/viewer/2022062919/5ee0f063ad6a402d666bffbc/html5/thumbnails/16.jpg)
Kafka Processing Topologies
Mesos
Elastic Data Processing and Analytics Stack
Open REST API (Play)
Pre-‐fabricated Hardware
Applications
HDFS
Service Discovery
Marathon StormChronos
Accumulo Monitoring
![Page 17: Mesos%@Bloomberg · InfluxDB Grafana Monitoring%Applications. Mesos Slave Log Shipper App1 App2 App3 StatsD CollectD Kafka Master Master Master ELK InfluxDB Grafana Monitoring%Applications.](https://reader035.fdocuments.net/reader035/viewer/2022062919/5ee0f063ad6a402d666bffbc/html5/thumbnails/17.jpg)
Marathon
Slave
Proxy Bridge
HA Proxy
SVC 1 SVC 2
Slave
Proxy Bridge
HA Proxy
SVC 1 SVC 3
Service Discovery (Mesosphere Implementation)
• Only works with Marathon • No support for multiple Marathons • Services not running on Mesos • How do clients discover global port in Marathon? • How do external clients discover services running on Mesos?
![Page 18: Mesos%@Bloomberg · InfluxDB Grafana Monitoring%Applications. Mesos Slave Log Shipper App1 App2 App3 StatsD CollectD Kafka Master Master Master ELK InfluxDB Grafana Monitoring%Applications.](https://reader035.fdocuments.net/reader035/viewer/2022062919/5ee0f063ad6a402d666bffbc/html5/thumbnails/18.jpg)
Marathon Marathon HDFS Zookeeper
SD Python Endpoint SD REST Endpoint
Client
Slave
Proxy Bridge
HA Proxy
SVC 1 SVC 2
Slave
Proxy Bridge
HA Proxy
SVC 1 SVC 3
Service Discovery Modifications
DNS
![Page 19: Mesos%@Bloomberg · InfluxDB Grafana Monitoring%Applications. Mesos Slave Log Shipper App1 App2 App3 StatsD CollectD Kafka Master Master Master ELK InfluxDB Grafana Monitoring%Applications.](https://reader035.fdocuments.net/reader035/viewer/2022062919/5ee0f063ad6a402d666bffbc/html5/thumbnails/19.jpg)
Master Master
Monitoring Applications
Master
![Page 20: Mesos%@Bloomberg · InfluxDB Grafana Monitoring%Applications. Mesos Slave Log Shipper App1 App2 App3 StatsD CollectD Kafka Master Master Master ELK InfluxDB Grafana Monitoring%Applications.](https://reader035.fdocuments.net/reader035/viewer/2022062919/5ee0f063ad6a402d666bffbc/html5/thumbnails/20.jpg)
Master Master Master
Monitoring Applications
![Page 21: Mesos%@Bloomberg · InfluxDB Grafana Monitoring%Applications. Mesos Slave Log Shipper App1 App2 App3 StatsD CollectD Kafka Master Master Master ELK InfluxDB Grafana Monitoring%Applications.](https://reader035.fdocuments.net/reader035/viewer/2022062919/5ee0f063ad6a402d666bffbc/html5/thumbnails/21.jpg)
Monitoring Applications
Master Master Master
• Log aggregation • Application statistics • Alerting!
![Page 22: Mesos%@Bloomberg · InfluxDB Grafana Monitoring%Applications. Mesos Slave Log Shipper App1 App2 App3 StatsD CollectD Kafka Master Master Master ELK InfluxDB Grafana Monitoring%Applications.](https://reader035.fdocuments.net/reader035/viewer/2022062919/5ee0f063ad6a402d666bffbc/html5/thumbnails/22.jpg)
Mesos Slave
Log Shipper
App1 App2 App3
StatsD
CollectD
Master Master Master
Monitoring Applications
![Page 23: Mesos%@Bloomberg · InfluxDB Grafana Monitoring%Applications. Mesos Slave Log Shipper App1 App2 App3 StatsD CollectD Kafka Master Master Master ELK InfluxDB Grafana Monitoring%Applications.](https://reader035.fdocuments.net/reader035/viewer/2022062919/5ee0f063ad6a402d666bffbc/html5/thumbnails/23.jpg)
Mesos Slave
Log Shipper
App1 App2 App3
StatsD
CollectD
Master Master Master
Monitoring Applications
![Page 24: Mesos%@Bloomberg · InfluxDB Grafana Monitoring%Applications. Mesos Slave Log Shipper App1 App2 App3 StatsD CollectD Kafka Master Master Master ELK InfluxDB Grafana Monitoring%Applications.](https://reader035.fdocuments.net/reader035/viewer/2022062919/5ee0f063ad6a402d666bffbc/html5/thumbnails/24.jpg)
Mesos Slave
Log Shipper
App1 App2 App3
StatsD
CollectD
Kafka
Master Master Master
Monitoring Applications
![Page 25: Mesos%@Bloomberg · InfluxDB Grafana Monitoring%Applications. Mesos Slave Log Shipper App1 App2 App3 StatsD CollectD Kafka Master Master Master ELK InfluxDB Grafana Monitoring%Applications.](https://reader035.fdocuments.net/reader035/viewer/2022062919/5ee0f063ad6a402d666bffbc/html5/thumbnails/25.jpg)
Mesos Slave
Log Shipper
App1 App2 App3
StatsD
CollectD
Kafka
Master Master Master
ELK
Monitoring Applications
![Page 26: Mesos%@Bloomberg · InfluxDB Grafana Monitoring%Applications. Mesos Slave Log Shipper App1 App2 App3 StatsD CollectD Kafka Master Master Master ELK InfluxDB Grafana Monitoring%Applications.](https://reader035.fdocuments.net/reader035/viewer/2022062919/5ee0f063ad6a402d666bffbc/html5/thumbnails/26.jpg)
Mesos Slave
Log Shipper
App1 App2 App3
StatsD
CollectD
Kafka
Master Master Master
ELK
Monitoring Applications
![Page 27: Mesos%@Bloomberg · InfluxDB Grafana Monitoring%Applications. Mesos Slave Log Shipper App1 App2 App3 StatsD CollectD Kafka Master Master Master ELK InfluxDB Grafana Monitoring%Applications.](https://reader035.fdocuments.net/reader035/viewer/2022062919/5ee0f063ad6a402d666bffbc/html5/thumbnails/27.jpg)
Mesos Slave
Log Shipper
App1 App2 App3
StatsD
CollectD
Kafka
Master Master Master
ELK
InfluxDB Grafana
Monitoring Applications
![Page 28: Mesos%@Bloomberg · InfluxDB Grafana Monitoring%Applications. Mesos Slave Log Shipper App1 App2 App3 StatsD CollectD Kafka Master Master Master ELK InfluxDB Grafana Monitoring%Applications.](https://reader035.fdocuments.net/reader035/viewer/2022062919/5ee0f063ad6a402d666bffbc/html5/thumbnails/28.jpg)
Mesos Slave
Log Shipper
App1 App2 App3
StatsD
CollectD
Kafka
Master Master Master
ELK
InfluxDB Grafana
Monitoring Applications
![Page 29: Mesos%@Bloomberg · InfluxDB Grafana Monitoring%Applications. Mesos Slave Log Shipper App1 App2 App3 StatsD CollectD Kafka Master Master Master ELK InfluxDB Grafana Monitoring%Applications.](https://reader035.fdocuments.net/reader035/viewer/2022062919/5ee0f063ad6a402d666bffbc/html5/thumbnails/29.jpg)
Mesos Slave
Log Shipper
App1 App2 App3
StatsD
CollectD
Kafka
Master Master Master
ELK
InfluxDB Grafana
Riemann
Alerting
Monitoring Applications
![Page 30: Mesos%@Bloomberg · InfluxDB Grafana Monitoring%Applications. Mesos Slave Log Shipper App1 App2 App3 StatsD CollectD Kafka Master Master Master ELK InfluxDB Grafana Monitoring%Applications.](https://reader035.fdocuments.net/reader035/viewer/2022062919/5ee0f063ad6a402d666bffbc/html5/thumbnails/30.jpg)
Access Control
• Applications deployed via Marathon • Give the power to users to deploy when they want and what they want
• Isolate core services from accidents • Isolate user applications
Marathon
Kafka
Accumulo
Policy Engine
![Page 31: Mesos%@Bloomberg · InfluxDB Grafana Monitoring%Applications. Mesos Slave Log Shipper App1 App2 App3 StatsD CollectD Kafka Master Master Master ELK InfluxDB Grafana Monitoring%Applications.](https://reader035.fdocuments.net/reader035/viewer/2022062919/5ee0f063ad6a402d666bffbc/html5/thumbnails/31.jpg)
Access Control -‐ Deploying Application
Password W/ SSL
MarathonReverse Proxy
Key Store
1. Launch App
2. Store AppKey
3. Launch App
4. AppKey
![Page 32: Mesos%@Bloomberg · InfluxDB Grafana Monitoring%Applications. Mesos Slave Log Shipper App1 App2 App3 StatsD CollectD Kafka Master Master Master ELK InfluxDB Grafana Monitoring%Applications.](https://reader035.fdocuments.net/reader035/viewer/2022062919/5ee0f063ad6a402d666bffbc/html5/thumbnails/32.jpg)
Access Control
Password W/ SSL
MarathonReverse Proxy
Key Store
1. Update <AppKey>
2. Check Access <AppKey>
3. Update App
4. Success / Failure
![Page 33: Mesos%@Bloomberg · InfluxDB Grafana Monitoring%Applications. Mesos Slave Log Shipper App1 App2 App3 StatsD CollectD Kafka Master Master Master ELK InfluxDB Grafana Monitoring%Applications.](https://reader035.fdocuments.net/reader035/viewer/2022062919/5ee0f063ad6a402d666bffbc/html5/thumbnails/33.jpg)
Access Control
Password W/ SSL
Marathon
Reverse Proxy
Key Store
Manage Apps
NimbusManage Topologies
![Page 34: Mesos%@Bloomberg · InfluxDB Grafana Monitoring%Applications. Mesos Slave Log Shipper App1 App2 App3 StatsD CollectD Kafka Master Master Master ELK InfluxDB Grafana Monitoring%Applications.](https://reader035.fdocuments.net/reader035/viewer/2022062919/5ee0f063ad6a402d666bffbc/html5/thumbnails/34.jpg)
Lessons
• Protect your Zookeeper cluster(s) • Don’t run HDFS on Mesos in your first deployment • Understand the back-‐off factor in Marathon – https://github.com/mesosphere/marathon/issues/1504
• Clean up the sandboxes periodically (and frequently) • Build a monitoring infrastructure for applications • Run multiple clusters • Mesos is very stable!