Debugging Docker ContainersTroy Folger, ISS Inc • 04.27.2016

Docker!The Brave New World of Containers

● Microservice approach- faster delivery- granular configuration and deployment- scalability, reliability, maintainability

● Image layering

● Container security- sandboxing / process protection- filesystem abstraction / mapping

● Platform independence

Features of Docker Containers

Docker!

Oh, wait ...

Debugging Challenges

Debugging Approaches

Contributions / Questions

● Indirect access to container processes and files

● Image layering (inheritance) can obscure issues in certain situations

● Inter-container communication- network implementations- varying, changing methods

● Platform differences (subtle and not-so-subtle)

● Loss of functionality / change of established practices“Who moved my cheese?”

Docker Debugging Challenges

General understanding of various container debugging approaches may be the single most helpful thing to know.

A. Container issue emergesB. DismayC. Revisit candidate debugging approaches …D. Google, baby, GoogleE. …F. Profit!

Docker Debugging Approaches

First rule: learn and apply docker image building and container running “best practices”.

Including, but not limited too:

● Microservice philosophy - one process / service per container● Simple Dockerfiles - use image layers to your advantage!● Consistent build approach● Understand volume mounting and use appropriately (!)● Understand and utilize “industry-standard” tools and practices (docker-compose,

docker-machine, etc)

Docker Debugging Approaches

“Best practices”, continued:

● Keep deployment / environment information separate from docker images● Keep in mind that not all deployments / use cases are the same - avoid over-

designing, use an iterative approach● Putting a poorly-designed application in a docker image does not magically make

the application elegant and reliable● Learn to love the command-line

Docker Debugging Approaches

Eclipse / IntelliJ / Other IDE tools

● Wait for it … wait …

Google. Seriously, these tools and setups change so frequently, the vendor web site and stackoverflow.com are going to be consulted at some point, anyway.

Docker Debugging Approaches

docker logs

● Interactive:$ docker logs CONTAINER | less$ docker logs CONTAINER > logfile

● Follow:$ docker logs -f CONTAINER

● Default log driver: JSON

Docker Debugging Approaches

docker attach

● Attach:$ docker attach [OPTIONS] CONTAINER

● Only really useful for containers running formatted stdout applications (think ‘top’)● Detach from the process via the default `ctrl-p ctrl-q` sequence

Docker Debugging Approaches

docker exec

● Console inside the container:$ docker exec -it CONTAINER /bin/bash

● Useful for running scripts INSIDE a container - example:$ docker run --name "platform" -v /usr/local/outer_dir/certs:/opt/issinc/certs \ -v /etc/localtime:/etc/localtime "platform/example_container" \ /bin/bash -c "cd /opt/issinc/certs ; /opt/scripts/test_certs.sh \ ca/DigiCertCA.crt host.key host.pem keystore.jks \ password "*.2thedge.com" truststore.jks password CA"

Docker Debugging Approaches

docker events

● Start a separate console running ‘docker events’● Docker containers report the following events:

attach, commit, copy, create, destroy, die, exec_create, exec_start, export, kill, oom, pause, rename, resize, restart, start, stop, top, unpause, update

● Docker images report the following events:delete, import, pull, push, tag, untag

● Docker volumes report the following events:create, mount, unmount, destroy

● Docker networks report the following events:create, connect, disconnect, destroy

Docker Debugging Approaches

docker events (continued)

● Filtering - currently supported filters are:container (container=<name or id>)event (event=<event action>)image (image=<tag or id>)label (label=<key> or label=<key>=<value>)type (type=<container or image or volume or network>)volume (volume=<name or id>)network (network=<name or id>)

Docker Debugging Approaches

docker diff

● Inspect changes on a container's filesystem:$ docker run --name ubuntu ubuntu:14.04 /bin/touch /tmp/difftest$ docker diff ubuntu C /runA /run/secretsC /tmpA /tmp/difftest

Docker Debugging Approaches

docker stats

● Display a live stream of one or more containers' resource usage:console #1: $ docker run -it --name ubuntu ubuntu:14.04 watch -n 2 ls /console #2: $ docker stats ubuntuCONTAINER CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O2cfe1cb26035 0.38% 5.738 MB / 33.68 GB 0.02% 508 B / 508 B 3.49 MB / 0 B

Docker Debugging Approaches

Override the ENTRYPOINT

● Default ENTRYPOINT specified in Dockerfile● No ENTRYPOINT specification will cause the container to use ‘/bin/sh -c’● Usage:

$ docker run -d -p 80:80 --entrypoint /bin/sh CONTAINER

● Often useful to ‘exec’ a container and then attempt the default ENTRYPOINT command in the resulting shell:$ docker exec -it CONTAINER /bin/bash$ cd /scripts$ ./run_me.sh

Docker Debugging Approaches

Add new runtime options with CMD

● Dockerfile CMD parameters will be passed to the container ENTRYPOINT and interpreted by that command (assuming the ENTRYPOINT is a command that interprets command-line arguments)

● ‘docker run’ treats parameters after the CONTAINER specification as CMD arguments:$ docker run -d -p 80:80 --entrypoint /bin/sh CONTAINER option1 option2

Docker Debugging Approaches

docker top

● $ docker top CONTAINER

Docker Debugging Approaches

docker inspect

● $ docker inspect CONTAINER

● JSON output with details of running container: mount points, log file, env vars, ports, etc.● The format option is useful:

$ docker inspect --format='{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' $INSTANCE_ID

Docker Debugging Approaches

docker history

● docker history CONTAINERff4d8bf9cd6c 4 months ago /bin/sh -c #(nop) CMD ["/bin/sh" "-c" "/opt/s 0 B ef89804b623a 4 months ago /bin/sh -c chmod -R +x /opt/scripts 14.78 kB f940c4b6fe31 4 months ago /bin/sh -c #(nop) ADD dir:f0ee20b44c1a3df8c25 14.78 kB c79487890f41 8 months ago /bin/sh -c apt-get update && apt-get install 10.37 MB …

● Use the ‘--no-trunc’ flag to see the entire command line for each of the image layers

Docker Debugging Approaches

docker commit

● In object-oriented parlance, images are ‘classes’, containers are ‘instances’ of images

● Committing a docker container causes a docker image to be committed to the registry

● Often the quickest way to make configuration changes, or do artifact replacement (jars, wars, property files, go classes, python scripts, etc)

● THIS IS NOT A REPLACEMENT FOR SOURCE CONTROL!!! You’re welcome.

Docker Debugging Approaches

Pause / Unpause the container

● Usage:$ docker run -d --name=pausetest alpine /bin/sh -c “while true; do sleep 2; date; done”$ docker pause CONTAINER$ docker unpause CONTAINER

Docker Debugging Approaches

Docker Debugging Approaches

Using (Abusing) volume mount directories

● Facilitates file copying / modifying● Very useful for debugging web applications● Keep in mind that many applications will cache files at start-up, so the container

may need to be restarted to notice your change● Note that read-only volume mounts will limit what you can achieve with this

technique● Not limited to ‘text files’ - sockets, pipes, other dev nodes can be volume mounted ...

Docker Debugging Approaches

Abusing docker container mapped directories

● Direct examination of container file system● DANGER DANGER … this is definitely a very efficient way to corrupt your container,

potentially eliminating the possibility of any future run of the container’s image ever running successfully ...

Restarting / Rebuilding Docker Infrastructure

● Restart docker daemon● Restart host networking● Remove docker0 bridge interface● Clean (‘rm -rf’) docker run-time directories:

# systemctl stop docker# rm -rf /var/lib/docker# systemctl start docker

- WARNING - last resort. This will remove all images and containers!

Docker Debugging Approaches

Network sniffing tools, process trace tools

● Containers typically take on host /proc/sys● Volume mount individual /dev nodes● Network tools may need to watch bridge0 as well default host interface

Docker Debugging Approaches

Docker daemon debug options

● Docker docs: ‘Enable debug output: -D’● Keep this in mind - startup problems are often at the daemon level, not so much at

the container level

Docker Debugging Approaches

Scripts / Source Instrumentation

● YourKit● Zipkin● Custom log driver for containers● Docker-specific logging

Docker Debugging Approaches

Docker Debugging Approaches

Debug / Instrumentation Containers

● Various monitor containers (Google cAdvisor)● Go nuts - create a mongo debug container that you deploy, start and stop as

required, for example

https://docs.docker.com

https://medium.com/@betz.mark/ten-tips-for-debugging-docker-containers-cde4da841a1d

https://opencredo.com/debugging-java-applications-running-in-docker/

https://github.com/google/cadvisor

References