1 @Dynatrace

Application Quality Metrics for your Pipeline

Andreas (Andi) Grabner - @grabnerandi

Shift-Left Quality

Example of a “Bad” Web Deployment 282! Objects on that page9.68MB Page Size

8.8s Page Load Time

Most objects are images delivered from your main

domain

Very long Connect time (1.8s) to your CDN

Example of a Bad Java/Tomcat Deployment526s to render a financial transaction report

1 SQL running 210s!

Debug Logging with log4j on outdated log4j

library (sync issue)

700 Deployments / Year

50-60 Deployments / Day

10+ Deployments / Day

Every 11.6 seconds

Challenges

Deploy Faster!!

Fail Faster!?

Its not about blind automation of pushing more bad code through a shiny pipeline

Metrics based

Decisions!

Time of D

eployment

Availability dropped to 0%

Usage by Channel? Errors on Devices?

Technical Debt!

80%$60B

Insufficient Focus on Quality

The “War Room”

Facebook – December 2012

20%80%

I learning from

others

4 use cases WHY did it happen? HOW to avoid it! METRICS to guide you.

26 @Dynatrace

27 @Dynatrace

#1don't push

without a plan

28 @Dynatrace

Mobile Landing Page of Super Bowl Ad

434 Resources in total on that page:230 JPEGs, 75 PNGs, 50 GIFs, …

Total size of ~ 20MB

29 @Dynatrace

Key Metrics# ResourcesSize of ResourcesPage Size

30 @Dynatrace

31 @Dynatrace

#2There is no easy "Migration" to Micro(Services)

32 @Dynatrace

26.7s Execution Time 33! Calls to the

same Web Service

171! SQL Queries through LINQ by this Web Service – request

similar data for each call

Architecture Violation: Direct access to DB instead from frontend logic

33 @Dynatrace

Key Metrics# Service Calls# of ThreadsSync and Wait Times# SQL executions# of SAME SQL’s

34 @Dynatrace

35 @Dynatrace

#3don't ASSUME you

know the environment

Distance calculation issues

480km biking in 1 hour!

Solution: Unit Test in Live App reports Geo

Calc Problems

Finding: Only happens on certain

Android versions

3rd party issues

Impact of bad 3rd party calls

38 @Dynatrace

Key Metrics# of functional errors# and Status of 3rd party callsPayload of Calls

12 000 000 $

40 @Dynatrace

#4Thinking Big?

Then Start Small!

41 @DynatraceAvailability dropped to 0%

Load Spike resulted in UnavailabilityAd on air

42 @Dynatrace

Alternative: “GoDaddy goes DevOps”

Response time improved 4x

1h before SuperBowl KickOff

1h after Game ended

43 @Dynatrace

Key Metrics

# Domains

Total Size of Content

44 @Dynatrace

What have we learned so far?

45 @Dynatrace

1. # Resources2. Size of Resources3. Page Size4. # Functional Errors5. 3rd Party calls6. # SQL Executions7. # of SAME SQLs

MetricBased

DecisionsAre Cool

We want to get from here …

To here!

Use these application metrics as additional Quality Gates

Extend your Continuous Integration

12 0 120ms3 1 68ms

Build 20 testPurchase OKtestSearch OK

Build 17 testPurchase OKtestSearch OK

Build 18 testPurchase FAILEDtestSearch OK

Build 19 testPurchase OKtestSearch OK

Build # Test Case Status # SQL # Excep CPU

12 0 120ms3 1 68ms

12 5 60ms3 1 68ms

75 0 230ms3 1 68ms

Test & Monitoring Framework Results Architectural Data

We identified a regresesion

Problem solved

Exceptions probably reason for failed testsProblem fixed but now we have an

architectural regressionProblem fixed but now we have an

architectural regressionNow we have the functional and architectural confidence

Let’s look behind the scenes

#1: Analyzing each Test

#2: Metrics for each Test

#3: Detecting Regression based on Measure

Quality-Metrics based Build Status

Pull data into Jenkins, Bamboo ...

Recap!

#1: Pick your App Metrics

# of Service Calls Bytes Sent & Received

# of Worker Threads

# of Worker Threads

# of SQL Calls, # of Same SQLs # of DB

Connections

# of SQL Calls, # of Same SQLs # of DB

Connections

#2: Figure out how to monitor them

#3: Automate it into your Pipeline

#4: Integrate with your Tools

Better Software,

Faster!!

Draw better Unicorns

60 @Dynatrace

Questions and/or DemoSlides: slideshare.net/grabnerandiGet Tools: bit.ly/dttrialYouTube Tutorials: bit.ly/dttutorialsContact Me: agrabner@dynatrace.comFollow Me: @grabnerandiRead More: blog.dynatrace.com

61 @Dynatrace

Andreas GrabnerDynatrace Developer Advocate@grabnerandihttp://blog.dynatrace.com