Preetam JinkaSoftware EngineerVividCortex@PreetamJinka

Matt WilliamsDevOps Evangelist Datadog@technovangelist

#WORKCENTRIC

Preetam JinkaSoftware Engineer

VividCortex is the best way to improve your database performance, efficiency, and uptime. It is a secure, cloud-hosted platform that eliminates your most critical APM visibility gap: deep insights into production database workload and query performance.

#WORKCENTRIC

Tip #1: Determine the work your systems are designed to performHow VividCortex views monitoring:

● Servers exist to do work.● Database work is in terms of queries.

We’re interested in:

● How databases complete their work● The behavior, efficiency, and effects of each query● How work changes over time

#WORKCENTRIC

Adaptive Fault DetectionDetecting micro-stalls in a database

#WORKCENTRIC

Database Stalls● A stall is a type of fault.● Short periods when work isn’t being done● Can be as short as 1 second● VividCortex detects real database stalls early before

they lead to user-facing outages or downtime● We do this with zero configuration and no fixed

thresholds◦ The secret sauce: we have a model.

#WORKCENTRIC

Tip #2: Find a model to create relationships between metrics or describe how work is being doneLittle’s Law

● L = λ × W● Concurrency = Throughput × Latency● Little’s Law provides a model to relate throughput and concurrency

In MySQL:

● Concurrency: threads_running◦ There’s one thread per query.◦ From SHOW STATUS

● Throughput: queries completed per second

#WORKCENTRIC

MySQL Server Stall

More queries in progress

Fewer being completed

#WORKCENTRIC

What about latency?We can see the effect of the stall on overall query latencies.

All of the stalled queries are completing after the fault ends.

#WORKCENTRIC

Monitoring QueriesLooking at the work itself

#WORKCENTRIC

“What gets measured gets managed.”

—Peter Drucker

#WORKCENTRIC

Tip #3: Monitor what you want to optimize

● Monitor anything worth optimizing.● If you’re interested in optimizing the use of indexes, monitor how queries

are using indexes.

#WORKCENTRIC

Work metrics Resource metrics

Tip #4: Focus on heavy hitters

#WORKCENTRIC

Monitoring query behavior over time

Detecting Workload Changes

#WORKCENTRIC

Tip #5:Automatically detect changesWhat’s taking up the database’s time that wasn’t before?

#WORKCENTRIC

Query Anomaly Detection● Detects changes in a query’s execution time, error rate, and

throughput● Uses intelligent baselining to account for seasonalities in metrics● Consider at least hourly, daily, and weekly seasonal trends● Not about detecting problems.

◦ Not worth alerting on!◦ Systems are always changing.

#WORKCENTRIC

Anomaly detected in a query’s total execution time metric

#WORKCENTRIC

Summary1. Determine the work your systems are designed to

perform.2. Find a model to create relationships between

metrics or describe how work is being done.3. Monitor what you want to optimize.4. Focus on the heavy hitters.5. Automatically detect changes.

https://www.vividcortex.com/blog/webinar-cheat-sheet

#WORKCENTRIC

Matt WilliamsDevOps Evangelist

#WORKCENTRIC

Datadog Overview

• SaaS based infrastructure monitoring• Focus on modern infrastructure

• Cloud, Containers, Micro Services• Processing nearly a trillion data points per day• Intelligent alerting, Insightful dashboardsnsightful Dashboards and

Reportin

#WORKCENTRIC

#WORKCENTRIC

Collecting data is cheap

#WORKCENTRIC

Collecting data is cheap

Not having it when you need it can be

expensive

#WORKCENTRIC

#WORKCENTRIC

#WORKCENTRIC

● Requests per second● % 404 Requests

#WORKCENTRIC

#WORKCENTRIC

● CPU Utilization● Queue Length

#WORKCENTRIC

#WORKCENTRIC

● Jenkins deploy● Git Commit

#WORKCENTRIC

#WORKCENTRIC

#WORKCENTRIC

DISCUSSIONPreetam & Matt

Request a Free Trialwww.vividcortex.com/free-trial-sign-up

app.datadoghq.com/signup

#WORKCENTRIC

Q&A Session

Additional Questions?preetam@vividcortex.com

matt.williams@datadoghq.com