Monitorama 2015 Monitoring OpenConnect CDN

Monitoring OpenConnect CDN

Sergey Fedorov, NetflixMonitorama 2015

Sergey Fedorov, Netflix, Monitorama 2015

What is OpenConnect

US downstream traffic *

* 2015 Sandvine reportSergey Fedorov, Netflix, Monitorama 2015

OpenConnect Cache Appliance

Space/Power optimized10/40Gbs network interfaceFreeBSD OSNGinx serverBird routing proxy

Gizmodo, “This box can hold an entire Netflix” http://gizmodo.com/this-box-can-hold-an-entire-netflix-1592590450Sergey Fedorov, Netflix, Monitorama 2015

Network

Transit

Internet Exchange

ISP embedded

Intelligent clients

Control Plane

end-user content request router

client locationnetwork conditionsserver utilizationcontent distribution

Who we are

Sergey Fedorov Stefan PraszalowiczSergey Fedorov, Netflix, Monitorama 2015

Monitoring challenge

Testing in prod*

Network changesFirmware deploymentsApp pushesUpdating content...

CachesClients

Control Plane

Microservices

Network

Capacity

Config

Content

Telemetry (Atlas)Logs (ElasticSearch)

Data sources

METRICS

Something breaks all the time

Big problems start small

Context matters

Small SRE team

Elastic

How we do it

Netflix Clients Caches Network ConfigData sources ......

Orchestration

Data processing

stream processorspollers

FSMState processing

Orchestration

Data processing

MAINTENANCE

start fixing end fixing

threshold=75%

action: okfrom: cpu

threshold=75%

MAINTENANCE

action: okfrom: cpu

threshold=75%

MAINTENANCE

action: okfrom: cpu

threshold=75%

MAINTENANCE

action: okfrom: cpu

threshold=75%

MAINTENANCE

action: silencefrom: config

threshold=75%

MAINTENANCE

action: okfrom: cpu

threshold=75%

MAINTENANCE

action: silencefrom: config

threshold=75%

MAINTENANCE

action: breakfrom: cpu

threshold=75%

MAINTENANCE

threshold=75%

MAINTENANCE

threshold=75%

MAINTENANCE

threshold=75%

MAINTENANCE

threshold=75%

MAINTENANCE

threshold=75%

MAINTENANCE

action: okfrom: cpu

threshold=75%

MAINTENANCE

action: unsilencefrom: config

threshold=75%

MAINTENANCE

action: okfrom: cpu

threshold=75%

MAINTENANCE

action: okfrom: cpu

threshold=75%

MAINTENANCE

action: okfrom: cpu

threshold=75%

MAINTENANCE

action: okfrom: cpu

threshold=75%

MAINTENANCE

threshold=75%

MAINTENANCE

threshold=75%

MAINTENANCE

action: start_fixfrom: user

threshold=75%

MAINTENANCE

threshold=75%

MAINTENANCE

threshold=75%

MAINTENANCE

threshold=75%

MAINTENANCE

threshold=75%

MAINTENANCE

threshold=75%

MAINTENANCE

threshold=75%

MAINTENANCE

threshold=75%

MAINTENANCE

action: okfrom: cpu

threshold=75%

MAINTENANCE

action: okfrom: cpu

threshold=75%

MAINTENANCE

action: end_fixfrom: user

threshold=75%

MAINTENANCE

threshold=75%

FSMState processing

Orchestration

Data processing

FSMState processing

Orchestration

Data processing

Events processingEvent handlers

STATE TRANSITION EVENT● OLD STATE● NEW STATE● Input action● Metric name● Action metadata

○ metric value○ comments○ tags○ timestamp○ ...

Event handlers

Triggers an event

Event handlersRULES

Events priority

Escalation

Do Never

Notice

Warning

Critical

Severity

Do Next

Do Last

Do Now

0 1 2 3

Notice

Warning

Critical

Severity

0 1 2 3Escalation

Notice

Warning

Critical

Severity

0 1 2 3

Notifications

FSMState processing

Orchestration

Data processing

Aggregation

ClusterCache state = aggregation of states of its metrics

Cluster state = aggregation of states of its caches

OK all OK DEGRADED some BROKEN or DEGRADEDBROKEN most BROKEN

All caches are OK → cluster state is OK

Aggregation

Cluster OK all OK DEGRADED some BROKEN or DEGRADEDBROKEN most BROKEN

2/12 caches are BROKEN → cluster state is DEGRADED

Aggregation

Cluster OK all OK DEGRADED some BROKEN or DEGRADEDBROKEN most BROKEN

7/12 caches are BROKEN → cluster state is BROKEN

FSMState processing

Orchestration

Data processing

Challenges

SetupPredefined groupings

Challenges

SetupPredefined groupingsUI

Challenges

SetupPredefined groupingsUIIssues correlation

Challenges

SetupPredefined groupingsUIIssues correlationFailure forecasting

Challenges

SetupPredefined groupingsUIIssues correlationFailure forecastingOSS

Feedback

jobs.netflix.com/jobs/1693/

jobs.netflix.com/jobs/2240/

Sergey FedorovOpenConnect, Netflixsfedorov@netflix.com

Monitorama 2015 Monitoring OpenConnect CDN

Technology

Transcript of Monitorama 2015 Monitoring OpenConnect CDN

Monitorama: 19 talks in 19 quotes

The Robots are Coming - SIG · 2018. 10. 2. · Kryon Systems NICE OpenConnect Pegasystems Redwood Software Softomotive Thoughtonomy UiPath WorkFusion Leaders ... The BPS provider

StatsD Workshop Monitorama 2013

Instructions - University of Toronto for Android...2/20/2016 OpenConnect for Android Twelve Secure your connection! forandroid/ 12/16

Monitorama PDX 2016 - Vizceral: Traffic Intuition

Car Alarms & Smoke Alarms [Monitorama]

5-2-Redes de distribucion de contenidos€¦ · Redes de distribución de contenidos (CDN) 4.1 CDN localización 4.2 CDN redirección Hora 2 4.3 CDN replicación 4.4 Estructura CDN

Stylehive Cdn

Dissertation CDN

Prometheus (Monitorama 2016)

Monitorama 2015 Netflix Instance Analysis

ÍNDICE - cdn.

cdn. · 2019. 9. 2. · cdn.

2-5 Installing the System into a Rack - oc.nflxvideo.netoc.nflxvideo.net/docs/OpenConnect-RevD_Railkits_2_and_4_posts.pdf2-4 SUPERSERVER 1027R-N3RF USER'S MANUAL 2-5 Installing the

Apresentação final referente ao subsistema monitoramento de bacias(PT-BAC), do projeto MONITORAMA Apresentação final referente ao subsistema monitoramento.

cdn-gd-v1.webbnc.net Mini...cdn-gd-v1.webbnc.net

Drupal CDN integration: easier, more ﬂexible and faster! · • Drupal + CDN (where we are) • Basic Drupal core patch • Simple CDN module • CDN integration module • Bachelor

VIRTUAL - cdn.

Metrics 2.0 @ Monitorama PDX 2014

Projeto MONITORAMA CMD-TD – Automação para a Tomada de Decisão