Monitoring 101 - Leveraging on the power of JMX
-
Upload
martin-goodwell -
Category
Software
-
view
227 -
download
2
Transcript of Monitoring 101 - Leveraging on the power of JMX
![Page 1: Monitoring 101 - Leveraging on the power of JMX](https://reader035.fdocuments.net/reader035/viewer/2022081604/58ae28241a28ab557e8b45cd/html5/thumbnails/1.jpg)
Monitoring 101Leverage on the Power of JMX
... and beyond
Martin GutenbrunnerDynatrace InnovationLab
@MartinGoodwell
Java Forum Stuttgart, July 7 #jfs2016
![Page 2: Monitoring 101 - Leveraging on the power of JMX](https://reader035.fdocuments.net/reader035/viewer/2022081604/58ae28241a28ab557e8b45cd/html5/thumbnails/2.jpg)
@MartinGoodwell
About me
Started with Commodore 8-bit (VC-20 and C-64) Built Null-Modem connections for playing Doom and
WarCraft I Went on to IPX/SPX networks between MS-DOS 6.22
and WfW 3.11 Did DevOps before it was a thing (mainly Java
and Web) for ~ 10 years Now at Dynatrace Innovation Lab Find me on Twitter: @MartinGoodwell Austria
Passionate about life, technology, and the people behind both of them.
![Page 3: Monitoring 101 - Leveraging on the power of JMX](https://reader035.fdocuments.net/reader035/viewer/2022081604/58ae28241a28ab557e8b45cd/html5/thumbnails/3.jpg)
@MartinGoodwell
Monitoring 101Leverage on the Power of JMX... and beyond
1
Agenda
2
3
![Page 4: Monitoring 101 - Leveraging on the power of JMX](https://reader035.fdocuments.net/reader035/viewer/2022081604/58ae28241a28ab557e8b45cd/html5/thumbnails/4.jpg)
@MartinGoodwell
Warm up
Please, feel free to ask and interrupt anytime This is about you, after all
How many developers, operators and business people are here? I love Java and Spring Framework
Anyone here who hates one of them? Any previous experience with JMX or monitoring, anyone?
Who knows what APM is?
![Page 5: Monitoring 101 - Leveraging on the power of JMX](https://reader035.fdocuments.net/reader035/viewer/2022081604/58ae28241a28ab557e8b45cd/html5/thumbnails/5.jpg)
@MartinGoodwell
Monitoring... your application, not your environment
![Page 6: Monitoring 101 - Leveraging on the power of JMX](https://reader035.fdocuments.net/reader035/viewer/2022081604/58ae28241a28ab557e8b45cd/html5/thumbnails/6.jpg)
@MartinGoodwell
Why monitoring, when we can debug? Debugging is for Developers only
Operations need monitoring Monitoring can be done via web apps
Debugging requires a dev-env to be setup Monitoring is about two things
Tracking problems Optimizing performance More often than not, those two things will be based on production data
Plus: when you can track errors, you can also track business
![Page 7: Monitoring 101 - Leveraging on the power of JMX](https://reader035.fdocuments.net/reader035/viewer/2022081604/58ae28241a28ab557e8b45cd/html5/thumbnails/7.jpg)
@MartinGoodwell
Active vs passive approachTwo ways of thinking, two ways of integrating
![Page 8: Monitoring 101 - Leveraging on the power of JMX](https://reader035.fdocuments.net/reader035/viewer/2022081604/58ae28241a28ab557e8b45cd/html5/thumbnails/8.jpg)
@MartinGoodwell
Active JMX is an active way of monitoring
You need to know, which metrics you want to monitor And actively publish/export them
Pro You can monitor all the metrics you want, even those specific to your application
Con Pollutes your code
But there‘s ways around that, like AOP
![Page 9: Monitoring 101 - Leveraging on the power of JMX](https://reader035.fdocuments.net/reader035/viewer/2022081604/58ae28241a28ab557e8b45cd/html5/thumbnails/9.jpg)
@MartinGoodwell
Passive
As opposed to passive monitoring Where tools automatically pick up the most common metrics, like
Number of requests Round-trip times HTTP status codes
by eg. AOP or Java agent interface I.e. without actively integrating monitoring code into your business logic
Pro: No code pollution
Con Only collects technology-related metrics (nr of requests, ...), no business metrics (like nr of orders)
![Page 10: Monitoring 101 - Leveraging on the power of JMX](https://reader035.fdocuments.net/reader035/viewer/2022081604/58ae28241a28ab557e8b45cd/html5/thumbnails/10.jpg)
@MartinGoodwell
JMXA short introduction
![Page 11: Monitoring 101 - Leveraging on the power of JMX](https://reader035.fdocuments.net/reader035/viewer/2022081604/58ae28241a28ab557e8b45cd/html5/thumbnails/11.jpg)
@MartinGoodwell
JMX Trivia Java Management Extensions Counters, gauges and strings JMX is ooold:
Integral part of Java since Java 5 MBeans can consist of
Readable/writable attributes (right, not only for reading values) Invokable operations (I am not really in favor of those) A description (rules of proper documentation apply)
![Page 12: Monitoring 101 - Leveraging on the power of JMX](https://reader035.fdocuments.net/reader035/viewer/2022081604/58ae28241a28ab557e8b45cd/html5/thumbnails/12.jpg)
@MartinGoodwell
Standard, Dynamic, Model, Open, MX MBean
Static Dynamic
ModelMBean (configurable) OpenMBean (specific data-types)
MXBean (came with Java 6) XMBean is not an MXBean with a typo
XMBean is a JBoss-specific MBean
Bottomline: use MXBean
![Page 13: Monitoring 101 - Leveraging on the power of JMX](https://reader035.fdocuments.net/reader035/viewer/2022081604/58ae28241a28ab557e8b45cd/html5/thumbnails/13.jpg)
@MartinGoodwell
Why JMX?
For Ops: Because most Java services / apps provide JMX metrics. You get it „for
free“ For Devs:
It‘s not too hard to implement, even really easy with eg. Spring Framework For DevOps (ie both):
Lots of visualization tools available, both free and commercial It‘s a unified way of monitoring (no matter whether it‘s a queue, a
database or a cache) For Business:
It allows your devs to provide and your ops to report the metrics you need.
![Page 14: Monitoring 101 - Leveraging on the power of JMX](https://reader035.fdocuments.net/reader035/viewer/2022081604/58ae28241a28ab557e8b45cd/html5/thumbnails/14.jpg)
@MartinGoodwell
Which metrics should I monitor?
Common process metrics CPU Memory
Service specific metrics Webservers: Nr of requests and response times Databases: Connection pools utilization, etc. Caches: hits and misses Queues: size, fill-rate
Business Intelligence Visitors Orders
![Page 15: Monitoring 101 - Leveraging on the power of JMX](https://reader035.fdocuments.net/reader035/viewer/2022081604/58ae28241a28ab557e8b45cd/html5/thumbnails/15.jpg)
@MartinGoodwell
Who provides JMX metrics out-of-the-box?
JVM Threading, Memory usage, Garbage Collection, CPU usage (system, process, …)
Web Servers Tomcat Jetty
Netflix OSS Eureka Discovery Server Hystrix Circuit Breaker
Connection Pools RDBMS (Tomcat DBCP, c3p0, BoneCP, HikariCP, …) MongoDB
Messaging Queues (ie JMS) HornetQ Active MQ
![Page 16: Monitoring 101 - Leveraging on the power of JMX](https://reader035.fdocuments.net/reader035/viewer/2022081604/58ae28241a28ab557e8b45cd/html5/thumbnails/16.jpg)
@MartinGoodwell
... Wait, there‘s still more!
Hibernate Spring
Adds JMX to most libraries it wraps Quartz (the scheduler) EhCache (the cache)
![Page 17: Monitoring 101 - Leveraging on the power of JMX](https://reader035.fdocuments.net/reader035/viewer/2022081604/58ae28241a28ab557e8b45cd/html5/thumbnails/17.jpg)
@MartinGoodwell
Some examples
![Page 18: Monitoring 101 - Leveraging on the power of JMX](https://reader035.fdocuments.net/reader035/viewer/2022081604/58ae28241a28ab557e8b45cd/html5/thumbnails/18.jpg)
@MartinGoodwell
Java JMX
![Page 19: Monitoring 101 - Leveraging on the power of JMX](https://reader035.fdocuments.net/reader035/viewer/2022081604/58ae28241a28ab557e8b45cd/html5/thumbnails/19.jpg)
@MartinGoodwell
Tomcat JMX
![Page 20: Monitoring 101 - Leveraging on the power of JMX](https://reader035.fdocuments.net/reader035/viewer/2022081604/58ae28241a28ab557e8b45cd/html5/thumbnails/20.jpg)
@MartinGoodwell
MongoDB JMX
![Page 21: Monitoring 101 - Leveraging on the power of JMX](https://reader035.fdocuments.net/reader035/viewer/2022081604/58ae28241a28ab557e8b45cd/html5/thumbnails/21.jpg)
@MartinGoodwell
HornetQ
![Page 22: Monitoring 101 - Leveraging on the power of JMX](https://reader035.fdocuments.net/reader035/viewer/2022081604/58ae28241a28ab557e8b45cd/html5/thumbnails/22.jpg)
@MartinGoodwell
Business metrics
Feature usage Number of placed orders Where do my customers come from? Track a customer‘s path:
Catalog Shopping Cart Checkout Payment Dropouts
![Page 23: Monitoring 101 - Leveraging on the power of JMX](https://reader035.fdocuments.net/reader035/viewer/2022081604/58ae28241a28ab557e8b45cd/html5/thumbnails/23.jpg)
@MartinGoodwell
Correlating metrics
On a technical basis Nr of requests vs response time
Increasing response time alongside increasing number of requests probably pinpoints a scalability problem
The real fun starts when you correlate BI with technical metrics Eg. Feature usage vs error rate or response times Increasing errors in service X
And decreasing usage of feature A at the same time There seems to be a relation
Order rate vs response time?
![Page 24: Monitoring 101 - Leveraging on the power of JMX](https://reader035.fdocuments.net/reader035/viewer/2022081604/58ae28241a28ab557e8b45cd/html5/thumbnails/24.jpg)
@MartinGoodwell
Correlating Business and Tech
Any error count makes a great candidate for correlation with most metrics
![Page 25: Monitoring 101 - Leveraging on the power of JMX](https://reader035.fdocuments.net/reader035/viewer/2022081604/58ae28241a28ab557e8b45cd/html5/thumbnails/25.jpg)
@MartinGoodwell
Build 17 testNewsAlert OKtestSearch OK
Build # Use Case Stat # API Calls # SQL Payload CPU
1 5 2kb 70ms1 3 5kb 120ms
Tests Metrics
Build 26 testNewsAlert OKtestSearch OK
Build 25 testNewsAlert OKtestSearch OK
1 4 1kb 60ms34 171 104kb 550ms
Ops#ServInst Usage RT
1 0.5% 7.2s
1 63% 5.2s
1 4 1kb 60ms2 3 10kb 150ms
1 0.2% 5.2s
5 75% 2.5s
Build 35 testNewsAlert -testSearch OK
- - - -2 3 10kb 150ms
- - -8 80% 2.0
s
Metrics from and for Dev(to)Ops
Re-architecture -> Performance Fixes
![Page 26: Monitoring 101 - Leveraging on the power of JMX](https://reader035.fdocuments.net/reader035/viewer/2022081604/58ae28241a28ab557e8b45cd/html5/thumbnails/26.jpg)
@MartinGoodwell
How to export your own metricsFor Ops: How to tell your Devs how to export their own metrics
![Page 27: Monitoring 101 - Leveraging on the power of JMX](https://reader035.fdocuments.net/reader035/viewer/2022081604/58ae28241a28ab557e8b45cd/html5/thumbnails/27.jpg)
@MartinGoodwellBy Donsez - self-made, CC BY-SA 3.0, https://en.wikipedia.org/w/index.php?curid=6721989
![Page 28: Monitoring 101 - Leveraging on the power of JMX](https://reader035.fdocuments.net/reader035/viewer/2022081604/58ae28241a28ab557e8b45cd/html5/thumbnails/28.jpg)
@MartinGoodwell
Vanilla JMX
https://docs.oracle.com/javase/tutorial/jmx/mbeans/standard.html
![Page 29: Monitoring 101 - Leveraging on the power of JMX](https://reader035.fdocuments.net/reader035/viewer/2022081604/58ae28241a28ab557e8b45cd/html5/thumbnails/29.jpg)
@MartinGoodwellhttps://docs.oracle.com/javase/tutorial/jmx/mbeans/standard.html
![Page 30: Monitoring 101 - Leveraging on the power of JMX](https://reader035.fdocuments.net/reader035/viewer/2022081604/58ae28241a28ab557e8b45cd/html5/thumbnails/30.jpg)
@MartinGoodwellhttps://docs.oracle.com/javase/tutorial/jmx/mbeans/standard.html
![Page 31: Monitoring 101 - Leveraging on the power of JMX](https://reader035.fdocuments.net/reader035/viewer/2022081604/58ae28241a28ab557e8b45cd/html5/thumbnails/31.jpg)
@MartinGoodwell
JMX with Spring Boot
Spring simplifies JMX by creating an MBean server automatically (as does eg Tomcat) less boilerplate code for registering your own MBeans
![Page 32: Monitoring 101 - Leveraging on the power of JMX](https://reader035.fdocuments.net/reader035/viewer/2022081604/58ae28241a28ab557e8b45cd/html5/thumbnails/32.jpg)
@MartinGoodwell
![Page 33: Monitoring 101 - Leveraging on the power of JMX](https://reader035.fdocuments.net/reader035/viewer/2022081604/58ae28241a28ab557e8b45cd/html5/thumbnails/33.jpg)
@MartinGoodwell
JMX Tools
![Page 34: Monitoring 101 - Leveraging on the power of JMX](https://reader035.fdocuments.net/reader035/viewer/2022081604/58ae28241a28ab557e8b45cd/html5/thumbnails/34.jpg)
@MartinGoodwell
JConsole, JVisualVM
Pro Comes with JVM
Con Once you quit, all data is lost
![Page 35: Monitoring 101 - Leveraging on the power of JMX](https://reader035.fdocuments.net/reader035/viewer/2022081604/58ae28241a28ab557e8b45cd/html5/thumbnails/35.jpg)
@MartinGoodwell
Nagios Core Pro
Allows you to combine all different types of metrics
Host Application
Con Very tedious to setup
Dedicated plugins for each technology
![Page 36: Monitoring 101 - Leveraging on the power of JMX](https://reader035.fdocuments.net/reader035/viewer/2022081604/58ae28241a28ab557e8b45cd/html5/thumbnails/36.jpg)
@MartinGoodwell
Dynatrace Allows you to combine all different types of
metrics Host Application
Zero-conf, auto-detection
![Page 37: Monitoring 101 - Leveraging on the power of JMX](https://reader035.fdocuments.net/reader035/viewer/2022081604/58ae28241a28ab557e8b45cd/html5/thumbnails/37.jpg)
@MartinGoodwell
Bridges (eg JMX to HTTP)
https://jolokia.org/reference/html/protocol.html
![Page 38: Monitoring 101 - Leveraging on the power of JMX](https://reader035.fdocuments.net/reader035/viewer/2022081604/58ae28241a28ab557e8b45cd/html5/thumbnails/38.jpg)
@MartinGoodwell
Downsides of JMX
For Java code only Finding the right spots for monitoring might require some iterations
Potential source of „Hell Breaks Loose“ Triggering methods out of context Changing configuration values in-memory only
![Page 39: Monitoring 101 - Leveraging on the power of JMX](https://reader035.fdocuments.net/reader035/viewer/2022081604/58ae28241a28ab557e8b45cd/html5/thumbnails/39.jpg)
@MartinGoodwell
Alternatives?
![Page 40: Monitoring 101 - Leveraging on the power of JMX](https://reader035.fdocuments.net/reader035/viewer/2022081604/58ae28241a28ab557e8b45cd/html5/thumbnails/40.jpg)
@MartinGoodwell
statsd real quick
http://www.slideshare.net/DatadogSlides/dev-opsdays-tokyo2013effectivestatsdmonitoring@MartinGoodwell
![Page 41: Monitoring 101 - Leveraging on the power of JMX](https://reader035.fdocuments.net/reader035/viewer/2022081604/58ae28241a28ab557e8b45cd/html5/thumbnails/41.jpg)
@MartinGoodwell
Java code calling statsd client
http://rick-hightower.blogspot.co.uk/2015/05/working-with-statsd-and-java.html
![Page 42: Monitoring 101 - Leveraging on the power of JMX](https://reader035.fdocuments.net/reader035/viewer/2022081604/58ae28241a28ab557e8b45cd/html5/thumbnails/42.jpg)
@MartinGoodwell
Why JMX over other alternatives?
You might be using it already for JVM-metrics (or Tomcat, etc) GC, CPU-usage, requests, etc
While still „polluting“ your code, interfaces allow for good structures Allows to interact with the application (although I wouldn‘t
recommend this) Notifications on a code-level (
https://docs.oracle.com/javase/tutorial/jmx/notifs/index.html) Comes with JVM – saves you from bloating small solutions with
dependencies
![Page 43: Monitoring 101 - Leveraging on the power of JMX](https://reader035.fdocuments.net/reader035/viewer/2022081604/58ae28241a28ab557e8b45cd/html5/thumbnails/43.jpg)
@MartinGoodwell
Shout outRight after lunch break at 14.30In Room Silcher-Saal
![Page 44: Monitoring 101 - Leveraging on the power of JMX](https://reader035.fdocuments.net/reader035/viewer/2022081604/58ae28241a28ab557e8b45cd/html5/thumbnails/44.jpg)
@MartinGoodwell
“Go then, there are other worlds than these.”— Jake Chambers, The Dark Tower
[email protected]@MartinGoodwellDynatrace Innovation Labhttp://blog.ruxit.comhttp://www.dynatrace.com
![Page 45: Monitoring 101 - Leveraging on the power of JMX](https://reader035.fdocuments.net/reader035/viewer/2022081604/58ae28241a28ab557e8b45cd/html5/thumbnails/45.jpg)
@MartinGoodwell
References
JMX-Technology homepage http://www.oracle.com/technetwork/java/javase/tech/javamanagement-140525.
html Monitoring and Managing the Java Platform using JMX technology
https://docs.oracle.com/javase/8/docs/technotes/guides/management/agent.html
Call-tracing and performance management in microservice environments http://www.slideshare.net/MartinGoodwell/performance-monitoring-and-call-trac
ing-in-microservice-environments Jolokia https://jolokia.org/ Nagios https://www.nagios.org/ statsd https://github.com/etsy/statsd/wiki Sleepless Dev http://rick-hightower.blogspot.co.uk