Observability in DevOps with Modern Distributed Applications

© 2017 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED.

Observability in DevOps with Modern Distributed Applications

Dave Mangot Head of Site Reliability Engineering, SolarWinds Cloud @davemangot

© 2017 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED.© 2017 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED.

About Me

• 20+ years in systems engineering, have led DevOps and operational maturity transformations at multiple companies

• Big companies: Cable & Wireless, Salesforce, SolarWinds

• Small Companies: Terracotta, Tagged, Librato

• International DevOps contributor

@davemangot

5/9/16 2

© 2017 SolarWinds Worldwide, LLC. All rights reserved.

SolarWinds company overview: Quick facts

• Founded in 1999, over 18 years in the industry

• Over 250,000 customers in 170 countries, including 425 of the Fortune 500®

• Global headquarters in Austin, Texas, with 2,300 employees worldwide

• SolarWinds was taken private by private equity investment firms Thoma Bravo and Silver Lake Partners in February 2016

3

© 2017 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED.4

but we wouldn’t be upset if you did :)

You do not need to purchase these solutions from SolarWinds!

Disclaimer:


DevOps

14


DevOps

15

DevOps is a collaboration between Development, Operations and other teams with the recognition that we are tasked with achieving common business goals.


The 3 Ways of DevOps

16http://itrevolution.com/the-three-ways-principles-underpinning-devops/


The 3 Ways of DevOps

17


Feedback

18


2nd Way of DevOps

The Second Way is about creating the right to left feedback loops. The goal of almost any process improvement initiative is to shorten and amplify feedback loops so necessary corrections can be continually made. The outcomes of the Second Way include understanding and responding to all customers, internal and external, shortening and amplifying all feedback loops, and embedding knowledge where we need it.

19http://itrevolution.com/the-three-ways-principles-underpinning-devops/


If it’s not monitored, it doesn’t exist


Uptime Monitoring

22


Uptime Monitoring

23


https://blog.gigaspaces.com/amazon-found-every-100ms-of-latency-cost-them-1-in-sales/

28

Latency matters. Amazon found every 100ms of latency cost them 1% in sales. Google found an extra .5 seconds in search page generation time dropped traffic by 20%.


You can’t improve what you don’t measure


Observability

30https://peter.bourgon.org/blog/2017/02/21/metrics-tracing-and-logging.html


Logging (events)

32


Logging

33

tail -f /var/log/messages


Application Development/Troubleshooting

36


Setup

37


Logging

38

• Application Development

• Troubleshooting

• Deployments

• Search Alerts


Shipping

39


Search Alerts

40


Papertrail

41


Metrics (aggregatable)

44


Infrastructure Metrics

45


Host Metrics

46


root 23706 1 0 Nov18 ? 00:00:00 /bin/sh -c start-stop-daemon --start --quiet --pidfile /var/run/appoptics/appoptics-snapteld.pid —chuid appoptics --exec /opt/appoptics/sbin/snapteld --oknodo --config /opt/appoptics/etc/config.yaml --pidfile /var/run/appoptics/appoptics-snapteld.pid

Host Metrics


Host Metrics

48


Application Metrics

49


Kafka

50


Custom Metrics

51


Metric Submission

52

api = librato.connect('email', 'token')

q = api.new_queue()

q.add('temperature', 22.1, tags={'location': 'downstairs'})

q.add('temperature', 23.1, tags={'location': 'upstairs'})

q.submit()


Custom Metrics

53


The 4 Golden Signals

54

● Latency

● Traffic

● Errors

● Saturation



55

Latency

The time it takes to service a request.



56

Traffic

How much demand is being placed on your system.



57

Errors

The rate of requests that fail either explicitly, implicitly, or by policy.



58

Saturation

How “full” your service is.


Tracing (request scoped)

59


Distributed Tracing

61

Web

App

Database


Distributed Tracing

62

Web

App

Database

X-Trace: 123654

X-Trace: 123654

X-Trace: 123654


WHAT IS DISTRIBUTED TRACING?

HELP!No, help ME!


export APPOPTICS_SERVICE_KEY="795fXXXXXXXe5f5b17c7:serviceA"

-javaagent:/usr/local/lib/appoptics-agent.jar

64


Distributed Tracing

65


Distributed Tracing

66


Distributed Tracing

67


Thanks!

70

@davemangot

Observability in DevOps with Modern Distributed Applications

Engineering

Transcript of Observability in DevOps with Modern Distributed Applications