Logs to AWS Cloudwatch from Docker Containers

Question

I have a few docker containers running with docker-compose on an AWS EC2 instance. I am looking to get the logs sent to AWS CloudWatch. I was also having issues getting the logs from docker containers to AWS CloudWatch from my Mac running Sierra so I've moved over to EC2 instances running Amazon AMI.

My docker-compose file:

version: '2'
services:
  scraper:
  build: ./Scraper/
  logging:
    driver: "awslogs"
    options:
       awslogs-region: "eu-west-1"
       awslogs-group: "permission-logs"
       awslogs-stream: "stream"
  volumes:
    - ./Scraper/spiders:/spiders

When I run docker-compose up I get the following error:

scraper_1 | WARNING: no logs are available with the 'awslogs' log driver

but the container is running. No logs appear on the AWS CloudWatch stream. I have assigned an IAM role to the EC2 container that the docker-containers run on.

I am at a complete loss now as to what I should be doing and would apprecaite any advice.

Are you running this on an ECS cluster or on plain EC2? – talentedmrjones Mar 22 '17 at 15:30 — talentedmrjones, Mar 22 '17 at 15:30
It is running on plain EC2. – user7692855 Mar 22 '17 at 15:30 — user7692855, Mar 22 '17 at 15:30

score 9 · Answer 1 · answered May 09 '17 at 11:30

The awslogs works without using ECS.

you need to configure the AWS credentials (the user should have IAM roles appropriate [cloudwatch logs]).

I used this tutorial, it worked for me: https://wdullaer.com/blog/2016/02/28/pass-credentials-to-the-awslogs-docker-logging-driver-on-ubuntu/

score 0 · Answer 2 · answered Nov 07 '18 at 00:48

I was getting the same error but when I checked the cloudwatch logs, I was able to see the logs in cloudwatch. Did you check that if you have the logs group created in cloudwatch. Docker doesn't support console logging when we use the custom logging drivers.

The section on limitations here says that docker logs command is only available for json-file and journald drivers, and that's true for built-in drivers.

When trying to get logs from a driver that doesn't support reading, nothing hangs for me, docker logs prints this:

Error response from daemon: configured logging driver does not support reading

score -6 · Answer 3 · answered Mar 22 '17 at 16:05

-6

The AWS logs driver you are using awslogs is for use with EC2 Container Service (ECS). It will not work on plain EC2. See documentation.

I would recommend creating a single node ECS cluster. Be sure the EC2 instance(s) in that cluster have a role, and the role provides permissions to write to Cloudwatch logs.

From there anything in your container that logs to stdout will be captured by the awslogs driver and streamed to Cloudwatch logs.

answered Mar 22 '17 at 16:05

talentedmrjones

7,511
1
26
26

There is nothing on https://docs.docker.com/engine/admin/logging/awslogs/#tag that shows it only works with the ECS. In fact, "or (if you are running the Docker daemon on an Amazon EC2 instance) the Amazon EC2 instance profile." would lead you to believe it works elsewhere to. – user7692855 Mar 22 '17 at 17:02
I wouldn't rely on Docker docs to be authoritative on the workings of AWS. Those log drivers in question if Im not mistaken rely on the ECS client running on EC2. In that regard you could *potentially* install the client and see if that helps – talentedmrjones Mar 22 '17 at 19:45
2

I'm using it on a regular EC2 instance, ECS is not required. – user2707671 Sep 12 '17 at 11:02
2

This answer should be removed, it leads to a wrong path for the most people that re struggling with this issue. – SEQOY Development Team Feb 04 '21 at 01:03

Logs to AWS Cloudwatch from Docker Containers

3 Answers3