Post on 21-Feb-2017
#Perform2017
with Heapster, Grafana, Fluentd, Elastic Stack, Jolokia and OpenTracingMartin Etmajer @metmajerDynatrace, Technology Lead
Monitoring, Logging and Tracing on Kubernetes
Wolfgang Amadeus Mozart1756 – 1791
The „Jay Z“ of classical music
Mike Grabner
Schwarzenegger
Schnitzel
#Perform20178
Monitoringwith Kubernetes Dashboard, Heapster, influxdb & Grafana
$ git clone https://github.com/kubernetes/dashboard.git$ kubectl create –f src/deploy/kubernetes-dashboard.yaml
Example: Deploy Kubernetes Dashboardlikely to be present in your kube-system already
collects resources usage, etc.
Heapster
heapster
runs in a Pod
heapster
Heapster
source
Kubernetes
REST client
sinks
influxdb
GCL
kafkaelasticsearch
GCM
graphite
KubernetesDashboard
via built-in client
REST API
heapster
Example: Heapster (standalone)
source
Kubernetes
REST client
sinks
influxdb
GCL
kafkaelasticsearch
GCM
graphite
KubernetesDashboard
via built-in client
REST API
Example: Deploy Heapster (standalone)
$ git clone https://github.com/kubernetes/heapster.git$ kubectl create –f heapster/deploy/kube-config/standalone/
heapster
Example: Heapster (with influxdb and Grafana)
source
Kubernetes
REST client
sinks
influxdb grafana
KubernetesDashboard
via built-in client
REST API
query
Example: Deploy Heapster (with influxdb and Grafana)
$ git clone https://github.com/kubernetes/heapster.git$ kubectl create –f heapster/deploy/kube-config/influxdb/
#Perform201723
Loggingwith Fluentd and Elastic Stack
continuous stream of log data for each container
Fluentd
runs oneach host
Example: Fluentd (for container logs to elasticsearch)
watch container logs
elasticsearch kibana
queryingestvia configuration
Example: Deploy Fluentd (for container logs to elasticsearch)
$ git clone https://github.com/kubernetes/kubernetes.git$ kubectl create –f kubernetes/cluster/addons/fluentd-elasticsearch/
#Perform201731
Monitoringwith Jolokia
Java Management Extensions (JMX)to manage and monitor applications
Source: https://en.wikipedia.org/wiki/Java_Management_Extensions
DynamicMBean MXBeanStandard
MBeanInstrumentationLevel
instrument resources
MBeanServerAgent Level
agent for registered MBeans
Connectors Adaptors
RemoteManagementLevel
provides access to MBeanServerRMI, HTTP/SOAP HTML, SNMP
Example: Jolokia (get list of accessible MBeans)
$ curl http://localhost:8080/jolokia/list
includes memory, threading, gc, etc.
Example: Jolokia (read memory usage data)
$ curl http://localhost:8080/jolokia/read/java.lang:type=Memory
name of MBean
Example: Jolokia (read threading data)
$ curl http://localhost:8080/jolokia/read/java.lang:type=Threading
name of MBean
#Perform201738
Tracingwith OpenTracing
a formally specified standard API for reporting
with various API implementations
Why distributed (end-to-end) tracing?
Source: https://research.google.com/pubs/pub36356.html
initiator
servicesremote calls
Why distributed (end-to-end) tracing?
Source: https://research.google.com/pubs/pub36356.html
initiator
servicesremote calls
What does the architecture look like?
Why distributed (end-to-end) tracing?
Source: https://research.google.com/pubs/pub36356.html
initiator
servicesremote calls
Where is most of the time spent?
Why distributed (end-to-end) tracing?
Source: https://research.google.com/pubs/pub36356.html
initiator
servicesremote calls
How are services invoked?
Why distributed (end-to-end) tracing?
Source: https://research.google.com/pubs/pub36356.html
initiator
servicesremote calls
Which services aren’t invoked?
#Perform201748
...and else?
Thank you.