Monitoring 101Leverage on the Power of JMX

... and beyond

Martin Gutenbrunner

Dynatrace InnovationLab

@MartinGoodwell

Java Forum Stuttgart, July 7 #jfs2016

@MartinGoodwell

About me

Started with Commodore 8-bit (VC-20 and C-64)

Built Null-Modem connections for playing Doom and WarCraft I

Went on to IPX/SPX networks between MS-DOS 6.22 and

WfW 3.11

Did DevOps before it was a thing (mainly Java and Web)

for ~ 10 years

Now at Dynatrace Innovation Lab

Find me on Twitter: @MartinGoodwell

Austria

Passionate about life, technology, and the people behind both of them.

@MartinGoodwell

Monitoring 101Leverage on the Power of JMX

... and beyond

1

Agenda

2

3

@MartinGoodwell

Warm up

Please, feel free to ask and interrupt anytime

This is about you, after all

How many developers, operators and business people are here?

I love Java and Spring Framework

Anyone here who hates one of them?

Any previous experience with JMX or monitoring, anyone?

Who knows what APM is?

@MartinGoodwell

Monitoring... your application, not your environment

@MartinGoodwell

Why monitoring, when we can debug?

Debugging is for Developers only

Operations need monitoring

Monitoring can be done via web apps

Debugging requires a dev-env to be setup

Monitoring is about two things

Tracking problems

Optimizing performance

More often than not, those two things will be based on production data

Plus: when you can track errors, you can also track business

@MartinGoodwell

Active vs passive approachTwo ways of thinking, two ways of integrating

@MartinGoodwell

Active

JMX is an active way of monitoring

You need to know, which metrics you want to monitor

And actively publish/export them

Pro

You can monitor all the metrics you want, even those specific to your application

Con

Pollutes your code

But there‘s ways around that, like AOP

@MartinGoodwell

Passive

As opposed to passive monitoring

Where tools automatically pick up the most common metrics, like

Number of requests

Round-trip times

HTTP status codes

by eg. AOP or Java agent interface

I.e. without actively integrating monitoring code into your business logic

Pro:

No code pollution

Con

Only collects technology-related metrics (nr of requests, ...), no business metrics (like nr of orders)

@MartinGoodwell

JMXA short introduction

@MartinGoodwell

JMX Trivia

Java Management Extensions

Counters, gauges and strings

JMX is ooold:

Integral part of Java since Java 5

MBeans can consist of

Readable/writable attributes (right, not only for reading values)

Invokable operations (I am not really in favor of those)

A description (rules of proper documentation apply)

@MartinGoodwell

Standard, Dynamic, Model, Open, MX

MBean

Static

Dynamic

ModelMBean (configurable)

OpenMBean (specific data-types)

MXBean (came with Java 6)

XMBean is not an MXBean with a typo

XMBean is a JBoss-specific MBean

Bottomline: use MXBean

@MartinGoodwell

Why JMX?

For Ops:

Because most Java services / apps provide JMX metrics. You get it „for free“

For Devs:

It‘s not too hard to implement, even really easy with eg. Spring Framework

For DevOps (ie both):

Lots of visualization tools available, both free and commercial

It‘s a unified way of monitoring (no matter whether it‘s a queue, a database or a

cache)

For Business:

It allows your devs to provide and your ops to report the metrics you need.

@MartinGoodwell

Which metrics should I monitor?

Common process metrics

CPU

Memory

Service specific metrics

Webservers: Nr of requests and response times

Databases: Connection pools utilization, etc.

Caches: hits and misses

Queues: size, fill-rate

Business Intelligence

Visitors

Orders

@MartinGoodwell

Who provides JMX metrics out-of-the-box?

JVM

Threading, Memory usage, Garbage Collection, CPU usage (system, process, …)

Web Servers

Tomcat

Jetty

Netflix OSS

Eureka Discovery Server

Hystrix Circuit Breaker

Connection Pools

RDBMS (Tomcat DBCP, c3p0, BoneCP, HikariCP, …)

MongoDB

Messaging Queues (ie JMS)

HornetQ

Active MQ

@MartinGoodwell

... Wait, there‘s still more!

Hibernate

Spring

Adds JMX to most libraries it wraps

Quartz (the scheduler)

EhCache (the cache)

@MartinGoodwell

Some examples

@MartinGoodwell

Java JMX

@MartinGoodwell

Tomcat JMX

@MartinGoodwell

MongoDB JMX

@MartinGoodwell

HornetQ

@MartinGoodwell

Business metrics

Feature usage

Number of placed orders

Where do my customers come from?

Track a customer‘s path:

Catalog

Shopping Cart

Checkout

Payment

Dropouts

@MartinGoodwell

Correlating metrics

On a technical basis

Nr of requests vs response time

Increasing response time alongside increasing number of requests probably pinpoints a

scalability problem

The real fun starts when you correlate BI with technical metrics

Eg. Feature usage vs error rate or response times

Increasing errors in service X

And decreasing usage of feature A at the same time

There seems to be a relation

Order rate vs response time?

@MartinGoodwell

Correlating Business and Tech

Any error count makes a great candidate for correlation with most metrics

@MartinGoodwell

Build 17 testNewsAlert OK

testSearch OK

Build # Use Case Stat # API Calls # SQL Payload CPU

1 5 2kb 70ms

1 3 5kb 120ms

Tests Metrics

Build 26 testNewsAlert OK

testSearch OK

Build 25 testNewsAlert OK

testSearch OK

1 4 1kb 60ms

34 171 104kb 550ms

Ops

#ServInst Usage RT

1 0.5% 7.2s

1 63% 5.2s

1 4 1kb 60ms

2 3 10kb 150ms

1 0.2% 5.2s

5 75% 2.5s

Build 35 testNewsAlert -

testSearch OK

- - - -

2 3 10kb 150ms

- - -

8 80% 2.0s

Metrics from and for Dev(to)Ops

Re-architecture -> Performance Fixes

@MartinGoodwell

How to export your own metricsFor Ops: How to tell your Devs how to export their own metrics

@MartinGoodwell

By Donsez - self-made, CC BY-SA 3.0, https://en.wikipedia.org/w/index.php?curid=6721989

@MartinGoodwell

Vanilla JMX

https://docs.oracle.com/javase/tutorial/jmx/mbeans/standard.html

@MartinGoodwellhttps://docs.oracle.com/javase/tutorial/jmx/mbeans/standard.html

@MartinGoodwellhttps://docs.oracle.com/javase/tutorial/jmx/mbeans/standard.html

@MartinGoodwell

JMX with Spring Boot

Spring simplifies JMX by

creating an MBean server automatically (as does eg Tomcat)

less boilerplate code for registering your own MBeans

@MartinGoodwell

@MartinGoodwell

JMX Tools

@MartinGoodwell

JConsole, JVisualVM

Pro

Comes with JVM

Con

Once you quit, all data is lost

@MartinGoodwell

Nagios Core Pro

Allows you to combine all different types of

metrics

Host

Application

Con

Very tedious to setup

Dedicated plugins for each technology

@MartinGoodwell

Dynatrace

Allows you to combine all different types of

metrics

Host

Application

Zero-conf, auto-detection

@MartinGoodwell

Bridges (eg JMX to HTTP)

https://jolokia.org/reference/html/protocol.html

@MartinGoodwell

Downsides of JMX

For Java code only

Finding the right spots for monitoring might require some iterations

Potential source of „Hell Breaks Loose“

Triggering methods out of context

Changing configuration values in-memory only

@MartinGoodwell

Alternatives?

@MartinGoodwell

statsd real quick

http://www.slideshare.net/DatadogSlides/dev-opsdays-tokyo2013effectivestatsdmonitoring

@MartinGoodwell

@MartinGoodwell

Java code calling statsd client

http://rick-hightower.blogspot.co.uk/2015/05/working-with-statsd-and-java.html

@MartinGoodwell

Why JMX over other alternatives?

You might be using it already for JVM-metrics (or Tomcat, etc)

GC, CPU-usage, requests, etc

While still „polluting“ your code, interfaces allow for good structures

Allows to interact with the application (although I wouldn‘t recommend this)

Notifications on a code-level

(https://docs.oracle.com/javase/tutorial/jmx/notifs/index.html)

Comes with JVM – saves you from bloating small solutions with dependencies

@MartinGoodwell

Shout out

Right after lunch break at 14.30

In Room Silcher-Saal

@MartinGoodwell

“Go then, there are other worlds than these.”

— Jake Chambers, The Dark Tower

martin.gutenbrunner@dynatrace.com

@MartinGoodwell

Dynatrace Innovation Lab

http://blog.ruxit.com

http://www.dynatrace.com

@MartinGoodwell

References

JMX-Technology homepage

http://www.oracle.com/technetwork/java/javase/tech/javamanagement-140525.html

Monitoring and Managing the Java Platform using JMX technology

https://docs.oracle.com/javase/8/docs/technotes/guides/management/agent.html

Call-tracing and performance management in microservice environments

http://www.slideshare.net/MartinGoodwell/performance-monitoring-and-call-tracing-in-microservice-environments

Jolokia https://jolokia.org/

Nagios https://www.nagios.org/

statsd https://github.com/etsy/statsd/wiki

Sleepless Dev http://rick-hightower.blogspot.co.uk