Monitoring Containers at Scale - September Webinar Series
-
Upload
amazon-web-services -
Category
Technology
-
view
768 -
download
4
Transcript of Monitoring Containers at Scale - September Webinar Series
© 2015, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Brandon Chavis, AWS Solutions ArchitectIlan Rabinovitch, Director of Technical Community, Datadog
20th September 2016
Monitoring Containers at Scale
Agenda
Introduction to ECS
AWS Logging options: Cloudwatch Logs, Cloudwatch, Cloudtrail
Logging containers with Datadog
Amazon EC2 Container Service (ECS)
Container Managementat Any Scale
Flexible ContainerPlacement
Integrationwith the AWS Platform
Components of Amazon ECS
TaskOne or more containers running together on an Instance
Task DefinitionDefinition of containers and environment configuration
ClusterFleet of EC2 instances on which tasks run
Cluster ManagerManage cluster resource and state of tasks
SchedulerPlaces tasks onto cluster
AgentCoordinate EC2 instances and Manager
Cluster, Scheduler, Task Scheduler
ManagerCluster
Task Definition
Task
Agent
Monitoring & Logging
AWS logging tools:
CloudwatchCloudwatch LogsCloudtrail
CloudWatch Logs with awslogs driver
Amazon CloudWatch Logs
Amazon CloudWatch Logs
Amazon CloudWatch Logs
Amazon CloudWatch Logs
Amazon S3
Amazon Kinesis
AWS Lambda
Amazon Elasticsearch Service
Amazon ECS Store
Stream
Process
Search
CloudWatch Logs driver
Configuring Logging in Task Definition
logConfiguration task definition parameter
Requires version 1.18 or greater of the Docker Remote API
Maps to docker run --log-driver option
Log drivers: json-file, syslog, journald, gelf, fluentd, awslogs
Configuring Logging in Task Definition
"containerDefinitions": [ { "memory": 300, "portMappings": [ {
"hostPort": 80, "containerPort": 80 } ],
"entryPoint": [ "sh", "-c" ], "logConfiguration": {
"logDriver": "awslogs", "options": {
"awslogs-group": "awslogs-test", "awslogs-region": "us-west-2", "awslogs-stream-prefix": "nginx" }
}, "name": "simple-app", "image": "httpd:2.4", "command": [ "/bin/sh -c \"echo 'Congratulations! Your application is now running on a container in Amazon ECS.'
> /usr/local/apache2/htdocs/index.html && httpd-foreground\"" ], "cpu": 10 } ], "family": "cw-logs-example"
}
Monitoring with Amazon CloudWatch
Metric data sent to CloudWatch in 1-minute periods and recorded for a period of two weeks
Available metrics: CPUReservation, MemoryReservation, CPUUtilization, MemoryUtilization
Available dimensions: ClusterName, ServiceName
Monitoring with Amazon CloudWatch
Monitoring with Amazon CloudWatch
Monitoring with Amazon CloudWatch
Use the Amazon CloudWatch Monitoring Scripts to monitor additional metrics, e.g. disk space:
# Edit crontab> crontab -e
# Add command to report disk space utilization to CloudWatch every five minutes*/5 * * * * <path_to>/mon-put-instance-data.pl --disk-space-util --disk-space-used --disk-space-avail --disk-path=/ --from-cron
Logging Amazon ECS API with AWS CloudTrail
{ "eventVersion": "1.03", "userIdentity": {…}, "eventTime": "2015-10-12T13:57:33Z", "eventSource": "ecs.amazonaws.com", "eventName": "CreateCluster", "awsRegion": "eu-west-1", "sourceIPAddress": "54.240.197.227", "userAgent": "console.amazonaws.com", "requestParameters": { "clusterName": "ecs-cli" },
Logging Amazon ECS API with AWS CloudTrail
"responseElements": { "cluster": { "clusterArn": "arn:aws:ecs:eu-west-1:560846014933:cluster/ecs-cli", "pendingTasksCount": 0, "registeredContainerInstancesCount": 0, "status": "ACTIVE", "runningTasksCount": 0, "clusterName": "ecs-cli", "activeServicesCount": 0 } }, […]
Monitoring Amazon ECS with Datadog
• SaaS based infrastructure and application monitoring• Focus on modern environments
• Cloud, Containers, Micro Services• Processing nearly a trillion data points per day• Intelligent Alerting and Insightful Dashboards
Datadog Overview
Operating Systems, Cloud Providers (AWS), Containers, Web Servers, Datastores, Caches, Queues and more...
Monitor Everything
CloudWatch and ECS
ResourcesCPUReservationMemoryReservationCPUUtilizationMemoryUtilization
How do we get at the upper layers?
Pseudo-files
• Provide visibility into container metrics via the file system. • Generally under: /cgroup/<resource>/docker/$CONTAINER_ID/ or/sys/fs/cgroup/<resource>/docker/$CONTAINER_ID/
Pseudo-files: CPU Metrics$ cat /sys/fs/cgroup/cpuacct/docker/$CONTAINER_ID/cpuacct.stat> user 2451 # time spent running processes since boot> system 966 # time spent executing system calls since boot
$ cat /sys/fs/cgroup/cpu/docker/$CONTAINER_ID/cpu.stat> nr_periods 565 # Number of enforcement intervals that have elapsed
> nr_throttled 559 # Number of times the group has been throttled
> throttled_time 12119585961 # Total time that members of the group were throttled (12.12 seconds)
Pseudo-files: CPU Throttling
Docker API• Detailed streaming metrics as JSON HTTP socket
$ curl -v --unix-socket /var/run/docker.sock http://localhost/containers/28d7a95f468e/stats
STATS Command
# Usage: docker stats CONTAINER [CONTAINER...]$ docker stats $CONTAINER_ID CONTAINER CPU % MEM USAGE/LIMIT MEM % NET I/O BLOCK I/Oecb37227ac84 0.12% 71.53 MiB/490 MiB 14.60% 900.2 MB/275.5 MB 266.8 MB/872.7 MB
Side Car Containers
Agents and Daemons
• Ideally we’d want to schedule an agent or daemon on each node via ECS Tasks.
• Current Solutions:1. Bake it into your image.2. Install on each host at provision time.3. Automate with User Scripts and Launch Configs
Grant Privileges via IAM$ aws iam create-role \ --role-name ecs-monitoring \ --assume-role-policy-document file://trust.policy
$ aws iam put-role-policy --role-name ecs-monitoring --policy-name ecs-monitoring-policy --policy-document file://ecs.policy
$ aws iam create-instance-profile --instance-profile-name ECSNode
$ aws iam add-role-to-instance-profile \ --instance-profile-name ECSNode \ --role-name ecs-monitoring
Create A User Script
Auto-Scale!
$ aws autoscaling create-launch-configuration --launch-configuration MyECSCluster --key-name my-key --image-id AMI_ID --instance-type INSTANCE_TYPE --user-data file://launch-script.txt --iam-instance-profile IAM_ROLE
Full Stack Monitoring
Docker API ECS & CloudWatch
Monitoring AgentContainer
Containers List &Metadata
Additional Metadata(Tags, events, etc)
Host Level Metrics
Monitoring Amazon ECS with Datadog
Aren’t we still missing a layer?
Operating Systems, Cloud Providers (AWS), Containers, Web Servers, Datastores, Caches, Queues and more...
Monitor Everything
© 2015, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Service Discovery
Service Discovery
Docker API ECS & CloudWatch
Monitoring AgentContainer
A O A O
Containers List &Metadata
Additional Metadata(Tags, etc)
Config Backend
Integration ConfigurationsHost Level Metrics
Custom Metrics
• Instrument custom applications
• You know your key transactions best.
• Use async protocols like STATSD
Demo
Monday, October 24, 2016 JW Marriot Austin
https://aws.amazon.com/events/devday-austin
Free, one-day developer event featuring tracks, labs, and workshops around Serverless,
Containers, IoT, and Mobile
Q&A If you want to learn more, register for our upcoming DevDay Austin: