Container logging: Best practices
Transcript of Container logging: Best practices
![Page 1: Container logging: Best practices](https://reader030.fdocuments.net/reader030/viewer/2022011813/61d4a5d28910641be34cb38f/html5/thumbnails/1.jpg)
![Page 2: Container logging: Best practices](https://reader030.fdocuments.net/reader030/viewer/2022011813/61d4a5d28910641be34cb38f/html5/thumbnails/2.jpg)
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Container logging: Best practices for consistency
Carmen Puccio
G P S T E C 3 2 9
Principal Solutions Architect
AWS Partner Program
Shashi Prabhakar
Manager, Partner Solutions Architects, Modernization & Linux
AWS Partner Program
![Page 3: Container logging: Best practices](https://reader030.fdocuments.net/reader030/viewer/2022011813/61d4a5d28910641be34cb38f/html5/thumbnails/3.jpg)
Before we begin, let’s categorize the logs
Typically developers are concerned with this because these logs give insight into application performance
• Example application: /var/log/myapp.log
• Web logs: journalctl -u nginx.service --since today
• Transaction logs
• Others
Typically operations teams are concerned with this because these logs give insight into
the stability of the platform and the availability of services
• Syslog and other OS logs
• Audit logs
• Performance metrics
![Page 4: Container logging: Best practices](https://reader030.fdocuments.net/reader030/viewer/2022011813/61d4a5d28910641be34cb38f/html5/thumbnails/4.jpg)
It’s all over the place
# Apache access log
127.0.0.1 - - [28/Jul/2019:10:22:04 -0300] "GET / HTTP/1.0" 200 2216
# Apache error log
127.0.0.1 - - [28/Jul/2019:10:27:32 -0300] "GET /hidden/ HTTP/1.0" 404 7218[Thu Jun 22 11:35:48 2019] [notice] caught SIGTERM, shutting down
# typical expressjs log output
[Mon, 21 Nov 2019 20:52:11 GMT] 200 GET /foo (1ms)
# /var/log/secure
Sep 27 16:48:13 ip-192-168-2-203 sshd[6181]: Invalid user super from 34.80.19.95 port 53100
# .NET core
Unhandled Exception: System.NullReferenceException: Object reference not set to an instance of an object.
at error.Program.Main(String[] args) in /Users/puccio/Desktop/myApp/Program.cs:line 11
![Page 5: Container logging: Best practices](https://reader030.fdocuments.net/reader030/viewer/2022011813/61d4a5d28910641be34cb38f/html5/thumbnails/5.jpg)
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
But you said you want to run containers?
![Page 6: Container logging: Best practices](https://reader030.fdocuments.net/reader030/viewer/2022011813/61d4a5d28910641be34cb38f/html5/thumbnails/6.jpg)
What about logging?
all the things!
Log
![Page 7: Container logging: Best practices](https://reader030.fdocuments.net/reader030/viewer/2022011813/61d4a5d28910641be34cb38f/html5/thumbnails/7.jpg)
What about logging? In reality
all the things . . .
almost
Log
![Page 8: Container logging: Best practices](https://reader030.fdocuments.net/reader030/viewer/2022011813/61d4a5d28910641be34cb38f/html5/thumbnails/8.jpg)
Application logging considerations• Implement a standard and easy-to-configure framework: Log4j2 / Log4net, NLog, Winston
• Log to STDOUT/STDERR
• If files are a must, think about the log names: mylog.yyyy-MM-dd HH:mm:ss.SSS.log
• Use logrotate or configure the docker-daemon.json
• {"log-driver":"json-file","log-opts":{"max-size":"10m","max-file":"3"}}
• Use log severity levels: DEBUG, INFO, WARN, and ERROR
• Use a standard structure such as JSON
• Custom formats and raw text logs need custom parsing rules in order to extract data for analysis
• Multiline messages are not your friend; dump them to JSON, even in this case
• This also applies to the schema for the fields. Use consistent names and label your services.
• {"time":"2019-11-04 12:34:56.0000","level":"ERROR","message":"HelloWorld","service":"foo","event":{"error":{"name":"RuntimeError","message":"MissingClass is undefined"}}}
• {"time":"2019-11-04 12:35:56.0000","level":"ERROR","message":"HelloWorld","service":"bar","event":{"error":{"exception":"RuntimeError","message":"MissingClass is undefined"}}}
• Use trace ID, for example, X-Amzn-Trace-Id for HTTP headers via the AWS X-Ray SDK
• X-Amzn-Trace-Id: Root=1-5759e988-bd862e3fe1be46a994272793;Sampled=1
![Page 9: Container logging: Best practices](https://reader030.fdocuments.net/reader030/viewer/2022011813/61d4a5d28910641be34cb38f/html5/thumbnails/9.jpg)
What just happened?
Frontend
service
AWS Fargate
Backend service
v2
AWS Fargate
Backend service
v1
AWS Fargate
Error
![Page 10: Container logging: Best practices](https://reader030.fdocuments.net/reader030/viewer/2022011813/61d4a5d28910641be34cb38f/html5/thumbnails/10.jpg)
Logging challenges with containers
• No permanent storage• Containers are not pets. If your container dies,
your logs will most likely be gone too.
• No fixed location from a network
address standpoint• You should not care about the IP addresses of
your containers
• No fixed location from a placement
perspective• Even though you can target nodes with things
such as label selectors, it’s all up to the
scheduler
![Page 11: Container logging: Best practices](https://reader030.fdocuments.net/reader030/viewer/2022011813/61d4a5d28910641be34cb38f/html5/thumbnails/11.jpg)
Log collection modes: Sidecar
app-container Sidecar
Core functionality Peripheral tasks
such as
• Logging
• Proxy
• Configuration
• Others
Instance
AWS Fargate
Amazon Elastic
Kubernetes Service
(Amazon EKS)
Amazon Elastic Container
Service (Amazon ECS)
![Page 12: Container logging: Best practices](https://reader030.fdocuments.net/reader030/viewer/2022011813/61d4a5d28910641be34cb38f/html5/thumbnails/12.jpg)
Log collection modes: DaemonSet
app-container
DaemonSet
Core functionality
Peripheral tasks
such as:
• Logging
• Node monitoring
• Others
Instance
app-container
DaemonSet
Core functionality
Peripheral tasks
such as:
• Logging
• Node monitoring
• Others
Instance
AWS Fargate Amazon EKSAmazon ECS
![Page 13: Container logging: Best practices](https://reader030.fdocuments.net/reader030/viewer/2022011813/61d4a5d28910641be34cb38f/html5/thumbnails/13.jpg)
DaemonSet versus SidecarWho is using what for logging?
DaemonSet Sidecar
Collecting logs Files Stdout & stderr/files
Deployment Standard manifest/deployment Standard manifest/deployment
Multitenant isolation Average? Strong?
ScaleRequests/limits
Scheduling conflicts?
Can lead to significant resource
consumption (requests/limits)
Resource usageLow (each node runs a
container)High (each pod runs a container)
Ability to customizeLow (1 DaemonSet covering all
nodes per node)
High (each pod is configured
separately)
Cluster size? Clusters with few functions? Large clusters? PaaS? SaaS?
Anything else? ? ?
![Page 14: Container logging: Best practices](https://reader030.fdocuments.net/reader030/viewer/2022011813/61d4a5d28910641be34cb38f/html5/thumbnails/14.jpg)
How to consolidate? Enter Fluent Bit
Amazon Elastic Container
Registry (Amazon ECR)
Amazon EKS
Amazon ECS
Fluent Bit
DaemonSet
Amazon Simple Storage
Service (Amazon S3)
Amazon Kinesis
Data FirehoseAmazon Athena
Fluent Bit
DaemonSet
app-container
app-container
![Page 15: Container logging: Best practices](https://reader030.fdocuments.net/reader030/viewer/2022011813/61d4a5d28910641be34cb38f/html5/thumbnails/15.jpg)
Related breakouts
CON309 FireLens
ANT312 Amazon Elasticsearch Service: Everything ingest
ANT332 Use Amazon ES to visualize and monitor containerized applications
![Page 16: Container logging: Best practices](https://reader030.fdocuments.net/reader030/viewer/2022011813/61d4a5d28910641be34cb38f/html5/thumbnails/16.jpg)
Thank you!
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Carmen Puccio
@CarmenAPuccio
Shashi Prabhakar
@shashikantprabhakar
![Page 17: Container logging: Best practices](https://reader030.fdocuments.net/reader030/viewer/2022011813/61d4a5d28910641be34cb38f/html5/thumbnails/17.jpg)
Please complete the session survey in the mobile app.
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
![Page 18: Container logging: Best practices](https://reader030.fdocuments.net/reader030/viewer/2022011813/61d4a5d28910641be34cb38f/html5/thumbnails/18.jpg)
Let’s start with the basics
$: docker run --name test -d busybox sh -c "while true; do $(echo date); sleep 1; done”
$: docker ps -aCONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
3bedb2d04f06 busybox "sh -c 'while true; …" 23 seconds ago Up 22 seconds test
$: docker logs test
Tue Sep 24 13:56:52 UTC 2019
Tue Sep 24 13:56:53 UTC 2019
Tue Sep 24 13:56:54 UTC 2019
Tue Sep 24 13:56:55 UTC 2019
Tue Sep 24 13:56:56 UTC 2019
![Page 19: Container logging: Best practices](https://reader030.fdocuments.net/reader030/viewer/2022011813/61d4a5d28910641be34cb38f/html5/thumbnails/19.jpg)
But what just happened?$: docker logs test
Tue Sep 24 13:56:52 UTC 2019
Tue Sep 24 13:56:53 UTC 2019
Tue Sep 24 13:56:54 UTC 2019
Tue Sep 24 13:56:55 UTC 2019
Tue Sep 24 13:56:56 UTC 2019
• Docker supports different logging drivers used to store and/or stream container stdout and stderr logs of the main container process (pid 1)
• The easiest and most used logging method for containerized applications is to write to the stdout and stderr streams
• By default, the Docker daemon uses the json-file logging driver
$: docker info --format '{{.LoggingDriver}}'
json-file
• When you start a container, you can configure it to use a different logging driver than the Docker daemon’s default, using the --log-driver flag
$: docker run --log-driver none --name test2 -d busybox sh -c "while true; do $(echo date); sleep 1; done”
$: docker inspect -f '{{.HostConfig.LogConfig.Type}}' test2
none
![Page 20: Container logging: Best practices](https://reader030.fdocuments.net/reader030/viewer/2022011813/61d4a5d28910641be34cb38f/html5/thumbnails/20.jpg)
Logging driversDriver Description
none No logs are available for the container, and docker logs do not return any output
local Logs are stored in a custom format designed for minimal overhead
json-file The logs are formatted as JSON, the default logging driver for Docker
syslog Writes logging messages to the syslog facility; the syslog daemon must be running on the host machine
journald Writes log messages to journald; the journald daemon must be running on the host machine
gelf Writes log messages to a Graylog extended log format (GELF) endpoint such as Graylog or Logstash
fluentd Writes log messages to fluentd (forward input); the fluentd daemon must be running on the host machine
awslogs Writes log messages to Amazon CloudWatch Logs
splunk Writes log messages to splunk using the HTTP Event Collector
etwlogs Writes log messages as event tracing for Windows (ETW) events (only available on Windows platforms)
gcplogs Writes log messages to Google cloud platform (GCP) logging
logentries Writes log messages to Rapid7 Logentries
![Page 21: Container logging: Best practices](https://reader030.fdocuments.net/reader030/viewer/2022011813/61d4a5d28910641be34cb38f/html5/thumbnails/21.jpg)
The awslogs log driver
$: docker run --log-driver=awslogs \
--log-opt awslogs-region=us-east-1 \
--log-opt awslogs-group=reinvent \
--log-opt awslogs-create-group=true \
--name test3 -d busybox sh -c "while true; do $(echo date); sleep 1; done"
![Page 22: Container logging: Best practices](https://reader030.fdocuments.net/reader030/viewer/2022011813/61d4a5d28910641be34cb38f/html5/thumbnails/22.jpg)
Amazon ECS & AWS Fargate log configuration
"logConfiguration": {
logDriver": "awslogs","fluentd","gelf","json-file","journald","logentries","splunk","syslog","awsfirelens",
"options": {
"string": "string"
...
},
"secretOptions": [{
"name": "string",
"valueFrom": "string"
}]
}
AWS FargateAmazon ECS
![Page 23: Container logging: Best practices](https://reader030.fdocuments.net/reader030/viewer/2022011813/61d4a5d28910641be34cb38f/html5/thumbnails/23.jpg)
/var/log/containers/POD-NAME_NAMESPACE_CONTAINER-NAME-CONTAINER-ID.log
What about Kubernetes?
• Logs should have a separate storage and lifecycle independent of nodes, pods, or containers
• This concept is called cluster-level logging
• Cluster-level logging requires a separate backend to store, analyze, and query logs
• Kubernetes provides no native storage solution for log data, but you can integrate many existing logging solutions into your Kubernetes cluster
stdout
stderr
app-container
my-pod
Instance
Deleted on pod eviction
![Page 24: Container logging: Best practices](https://reader030.fdocuments.net/reader030/viewer/2022011813/61d4a5d28910641be34cb38f/html5/thumbnails/24.jpg)
Fluent Bit on Amazon EKS
/var/log/containers/POD-NAME_NAMESPACE_CONTAINER-NAME-CONTAINER-ID.log
stdout
stderr
app-container
my-pod
Instance
Fluent Bit
DaemonSet
Read logs Look up
metadata
Kubernetes API Server
Analyze the tag and extract
the following metadata:• Pod name
• Namespace
• Container name
• Container ID
Query Kubernetes API Server in order to obtain extra metadata for the POD in question• Pod ID
• Labels
• Annotations