Running CloudWatch Agent in my Docker image on AWS ECS fails

Question

Understand for this question that I'm relatively new to Docker and AWS.

The goal is to create a single AWS ECS instance that runs Apache and PHP with a basic Laravel application. I want to run a CloudWatch agent to send all logs to CloudWatch (access and error logs for Apache, error log for PHP and the Laravel logs).

I know this probably isn't 'best practice' (tips are welcome), but my philosophy for now is 'first make it work, then make it pretty' :-)

My Dockerfile:

FROM amazonlinux:latest

# Update/Install
RUN yum update -y && \
    # Install PHP & epel
    amazon-linux-extras install -y php7.3 epel && \
    # Install
    yum install -y \
    # Install apache
    httpd \
    # Install tools for CloudWatch
    collectd statsd \
    # Install supervisor
    supervisor \
    # Install cloudwatch agent
    https://s3.amazonaws.com/amazoncloudwatch-agent/amazon_linux/amd64/latest/amazon-cloudwatch-agent.rpm && \
    # Clean install data
    yum clean metadata && \
    yum -y clean all && \
    rm -rf /var/cache/yum

# PHP Settings
RUN sed -i \
    '/<Directory \"\/var\/www\/html\">/,/<\/Directory>/ s/AllowOverride None/AllowOverride All/' \
    /etc/httpd/conf/httpd.conf

# Remove default html folder
RUN rm -rf /var/www/html

# Configure supervisor
COPY supervisord.conf /etc/supervisord.conf

# Configure CloudWatch agent
COPY amazon-cloudwatch-agent.json /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json

# Add source to image
ADD . /var/www/aws

RUN chown -R apache:apache /var/www && ln -s /var/www/aws/public /var/www/html

# Expose port 80
EXPOSE 80

# Start supervisor
CMD ["/usr/bin/supervisord", "-n", "-c", "/etc/supervisord.conf"]

My supervisor.conf

[supervisord]
nodaemon=true

[program:httpd]
priority=1
command=/usr/sbin/apachectl -D FOREGROUND
autorestart=true
username=apache

[program:php]
priority=2
command=/usr/sbin/php-fpm
autorestart=true

[program:cloudformation]
priority=10
command=/opt/aws/amazon-cloudwatch-agent/bin/start-amazon-cloudwatch-agent
autorestart=true

My cloudwatch config:

{
    "agent": {
        "metrics_collection_interval": 60,
        "region": "eu-europe-1",
        "logfile": "/opt/aws/amazon-cloudwatch-agent/logs/amazon-cloudwatch-agent.log",
        "debug": false,
        "run_as_user": "cwagent"
    },
    "logs": {
        "logs_collected": {
            "files": {
                "collect_list": [
                    {
                        "file_path": "/var/log/php-fpm/www-error.log",
                        "log_group_name": "aws-docker",
                        "log_stream_name": "{instance_id}"
                    }
                ]
            }
        }
    }
}

Basically this works fine as a docker image that runs the Laravel application. The only problem I'm having now is the CloudWatch agent. It starts on the container in ECS, but fails to run with the following message:

2020/02/22 13:39:28 I! 2020/02/22 13:39:28 E! ec2metadata is not available
I! Detected the instance is OnPrem
2020/02/22 13:39:28 Reading json config file path: /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json ...
Valid Json input schema.
I! Detecting runasuser...
2020/02/22 13:39:28 E! Credentials path is not set while runasuser is not root
2020/02/22 13:39:28 I! AmazonCloudWatchAgent Version 1.237768.0.
2020/02/22 13:39:28 Configuration validation first phase failed. Agent version: 1.237768.0. Verify the JSON input is only using features supported by this version.
2020/02/22 13:39:28 I! Return exit error: exit code=1
2020/02/22 13:39:28 E! Cannot translate JSON config into TOML, ERROR is exit status 1

First of all I don't understand why the message ec2metadata is not available is showing. The container runs on ECS, so it should be available (from what I understand).

Second the message Configuration validation first phase failed. Agent version: 1.237768.0. Verify the JSON input is only using features supported by this version.. As far as I can tell the config should be oke.

I think my role also is oke, because the container does send logs to CloudWatch.

What am I doing wrong?

add the env variable ECS_ENABLE_TASK_IAM_ROLE_NETWORK_HOST to be "true" — Lamanus, Feb 23 '20 at 15:14

score 5 · Answer 1 · answered May 27 '20 at 15:45

Here's how I bludgeoned it into submission. We're using a very lightweight Ubuntu subset in our Docker containers that doesn't have systemctl or System V init, which the CloudWatch Agent seems designed for. You can run start-amazon-cloudwatch-agent directly (as this answer mentioned), but it's not quite as smooth.

The agent wants to aggressively manage the owner/group of the /opt/aws/amazon-cloudwatch-agent tree. (a) If you start it as root with "runasuser": "cwagent" in the config, then it refuses to use the AWS config/creds (Credentials path is not set mentioned above), and it bails. (b) If you start it as cwagent with "runasuser": "cwagent", then it complains that it can't change the ownership of a bunch of stuff (even though it's already owned by cwagent), and it bails. (c) But if you start it as cwagent and don't include a "runasuser" in the config, it complains, but it does start up and do its thing.

The Verify the JSON input is only using features supported by this version. message is what the agent says whenever it runs into trouble. It seems to have nothing to do with the config (which is fine).

Here are my specifics:

# Dockerfile

ADD ./files /tmp
# [...]
RUN curl -o /tmp/amazon-cloudwatch-agent.deb 'https://s3.amazonaws.com/amazoncloudwatch-agent/ubuntu/amd64/latest/amazon-cloudwatch-agent.deb' \
    && dpkg -i /tmp/amazon-cloudwatch-agent.deb \
    && rm -f /tmp/amazon-cloudwatch-agent.deb \
    && usermod -a -G www-data cwagent \
    && chgrp -R www-data /var/log/nginx \
    && chmod g+s /var/log/nginx \
    && chown -R cwagent:cwagent /opt/aws/amazon-cloudwatch-agent \
    && install -o cwagent -g cwagent -m 700 -d /home/cwagent \
    && install -o cwagent -g cwagent -m 700 -d /home/cwagent/.aws \
    && install -o cwagent -g cwagent -m 600 /tmp/cloudwatch.config /home/cwagent/.aws/config \
    && install -o cwagent -g cwagent -m 600 /tmp/cloudwatch.credentials /home/cwagent/.aws/credentials \
    && install -o cwagent -g cwagent -m 755 /tmp/cloudwatch-agent.json /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.d/default \
    && mv /tmp/99_cloudwatch.init /etc/my_init.d/

# /etc/my_init.d/99_cloudwatch.init

#!/bin/sh
su cwagent -c "nohup /opt/aws/amazon-cloudwatch-agent/bin/start-amazon-cloudwatch-agent >/tmp/cwagent.out 2>&1 &"
exit 0

// cloudwatch-agent.json

{
  "agent": {
    "region": "us-east-1",
    "debug": false
  },
  "logs": {
    "logs_collected": {
      "files": {
        "collect_list": [
          {
            "file_path": "/var/log/nginx/access.log",
            "log_group_name": "our-app",
            "log_stream_name": "nginx-access",
            "timestamp_format": "[%d/%b/%Y:%H:%M:%S %z]"
          },
          {
            "file_path": "/var/log/nginx/error.log",
            "log_group_name": "our-app",
            "log_stream_name": "nginx-error",
            "timezone": "UTC",
            "timestamp_format": "%Y/%m/%d %H:%M:%S"
          }
        ]
      }
    }
  }
}

Running CloudWatch Agent in my Docker image on AWS ECS fails

1 Answers1

Linked