No logs appear on Cloudwatch log group for elastic beanstalk environment

Question

I have an elastic beanstalk environment, which is running a docker container that has a node js API. On the AWS Console, if I select my environment, then go to Configuration/Software I have the following:

Log groups: /aws/elasticbeanstalk/my-environment
Log streaming: Enabled
Retention: 3 days
Lifecycle: Keep after termination.

However, if I click on that log group on the Cloudwatch console, I have a Last Event Time of some weeks ago (which I believe corresponds to when the environment was created) and have no content on the logs.

Since this is a dockerized application, Logs for the server itself should be at /aws/elasticbeanstalk/my-environment/var/log/eb-docker/containers/eb-current-app/stdouterr.log. If I instead get the Logs directly from the instances by going once again to my EB environment, clicking "Logs" and then "Request last 100 Lines" the logging is happening correctly. I just can't see a thing when using CloudWatch.

Any help is gladly appreciated

Have you been redeploying your app ie: starting a new container? If so I think your question is related to https://stackoverflow.com/questions/57464585/elastic-beanstalk-log-path-not-updating-when-docker-container-changes — kierans, Sep 05 '19 at 13:27
Yes seems related, but unfortunately no answer there. If I check out my CloudWatch log groups I have something like : /aws/elasticbeanstalk/my-env/var/log/eb-docker/containers/eb-current-app/stdouterr.log However if I pull the logs directly I observe the following: /var/log/eb-docker/containers/eb-current-app/eb-"hexstring"-stdouterr.log — Farid Hajnal, Sep 06 '19 at 17:46

Farid Hajnal · Accepted Answer · 2019-09-17T15:49:53.213

I was able to get around this problem. So CloudWatch makes a hash based on the first line of your log file and the log stream key, and the problem is that my first line on the stdouterr.log file was actually an empty line!

After couple of days playing around and getting help from the good AWS support team, I first connected via SSH to my EC2 instance associated to the EB environment and you need to add the following line to the /etc/awslogs/config/beanstalklogs.conf file, right after the "file=/var/log/eb-docker/containers/eb-current-app/stdouterr.log" line:

file_fingerprint_lines=1-20

With these, you tell the AWS service that it should calculate the hash using lines 1 through 20 on the log file. You could change 20 for larger or smaller numbers depending on your logging content; however I don't know if there is an upper limit for the value.

After doing so, you need to restart the AWS Logs Service on the instance.

For this you would execute:

sudo service awslogs stop
sudo service awslogs start

or simpler:

sudo service awslogs restart

After these steps I started using my environment and the logging was now being properly streamed to the CloudWatch console! However this would not work if a new deployment is made, if the EC2 instance gets replaced or the auto scalable group spawns another.

To have a fix for this, it is possible to add log config via the .ebextensions directory, at the root of your application before deploying.

I added a file called logs.config to the newly created .ebextensions directory and placed the following content:

files:
  "/etc/awslogs/config/beanstalklogs.conf":
    mode: "000644"
    user: root
    group: root
    content: |
      [/var/log/eb-docker/containers/eb-current-app/stdouterr.log]
      log_group_name=/aws/elasticbeanstalk/EB-ENV-NAME/var/log/eb-docker/containers/eb-current-app/stdouterr.log
      log_stream_name={instance_id}
      file=/var/log/eb-docker/containers/eb-current-app/*stdouterr.log
      file_fingerprint_lines=1-20

commands:
  01_remove_eb_stream_config:
    command: 'rm /etc/awslogs/config/beanstalklogs.conf.bak'
  02_restart_log_agent:
    command: 'service awslogs restart'

Changing of course EB-ENV-NAME by my environment name on EB.

Hope it can help someone else!

Great job solving this as I also was having the same issues. Do your log statements contain timestamps? It was suggested to me to change my application logging config so log messages had a timestamp. Because any hashes will therefore always be different and you don't have to provide any extra Beanstalk logging configuration like you've done. — kierans, Sep 14 '19 at 06:35
@kierans that would also solve the problem from what I know. The thing is that I am using a framework that by default outputs the first line of the logs as an empty line, so I am guessing this was the one compared by the AWS logs agent, and is why I had to rely on modifying the "file_fingerprint_lines" — Farid Hajnal, Sep 16 '19 at 19:01
@FaridHajnal shouldn't this config remove the existing logging setup in beanstalklogs.conf ? — titus, Mar 27 '20 at 08:41
This did not work for my NodeJS Single Docker Instance (Docker running on 64bit Amazon Linux 2/3.0.3) . — Ariel Frischer, Jul 12 '20 at 08:51
@ArielFrischer hello. I am using NestJs framework for my server application, which is also running on a single docker instance environment — Farid Hajnal, Jul 13 '20 at 18:49
@Farid Hajnal Hi, do you have a question? Also I'd like to add to my previous script that it didn't work because it caused a deployment error. I'm currently trying some fixes with AWS support team, if something works I'll report as an answer below. — Ariel Frischer, Jul 13 '20 at 19:42

Matthew Buckett · Answer 2 · 2020-10-08T15:46:09.270

For 64 bit Amazon Linux 2 the setup is slightly different.

For the delivery of log the AWS CloudWatch Agent is installed in /opt/aws/amazon-cloudwatch-agent and the Elastic Beanstalk configuration is in /opt/aws/amazon-cloudwatch-agent/etc/beanstalk.json. It is set to log the output of the container assuming there's a file called stdouterr.log, here's a snippet of the config:

{
  "file_path": "/var/log/eb-docker/containers/eb-current-app/stdouterr.log",
  "log_group_name": "/aws/elasticbeanstalk/EB-ENV-NAME/var/log/eb-docker/containers/eb-current-app/stdouterr.log",
  "log_stream_name": "{instance_id}"
}

However when I look for the file_path it doesn't exist, instead I have a file path that encodes the current docker container ID /var/log/eb-docker/containers/eb-current-app/eb-e4e26c0bc464-stdouterr.log.

This logfile is created by a script /opt/elasticbeanstalk/config/private/eb-docker-log-start that is started by the eb-docker-log service, the default contents of this file are:

EB_CONFIG_DOCKER_CURRENT_APP=`cat /opt/elasticbeanstalk/deployment/.aws_beanstalk.current-container-id | cut -c 1-12`
mkdir -p /var/log/eb-docker/containers/eb-current-app/
docker logs -f $EB_CONFIG_DOCKER_CURRENT_APP >> /var/log/eb-docker/containers/eb-current-app/eb-$EB_CONFIG_DOCKER_CURRENT_APP-stdouterr.log 2>&1

To temporarily fix the logging you can manually run (replacing the docker ID) and then logs will start to appear in CloudWatch:

ln -sf /var/log/eb-docker/containers/eb-current-app/eb-e4e26c0bc464-stdouterr.log /var/log/eb-docker/containers/eb-current-app/stdouterr.log

To make this permanant I added an .ebextension to fix the eb-docker-log service so it re-makes this link so create a file in your source code in .ebextensions called fix-cloudwatch-logging.config and set it's contents to:

files:
  "/opt/elasticbeanstalk/config/private/eb-docker-log-start" :
    mode: "000755"
    owner: root
    group: root
    content: |
      EB_CONFIG_DOCKER_CURRENT_APP=`cat /opt/elasticbeanstalk/deployment/.aws_beanstalk.current-container-id | cut -c 1-12`
      mkdir -p /var/log/eb-docker/containers/eb-current-app/
      ln -sf /var/log/eb-docker/containers/eb-current-app/eb-$EB_CONFIG_DOCKER_CURRENT_APP-stdouterr.log /var/log/eb-docker/containers/eb-current-app/stdouterr.log
      docker logs -f $EB_CONFIG_DOCKER_CURRENT_APP >> /var/log/eb-docker/containers/eb-current-app/eb-$EB_CONFIG_DOCKER_CURRENT_APP-stdouterr.log 2>&1
commands:
  fix_logging:
    command: systemctl restart eb-docker-log.service
    cwd: /home/ec2-user
    test: "[ ! -L /var/log/eb-docker/containers/eb-current-app/stdouterr.log ] && systemctl is-active --quiet eb-docker-log"

This worked for me! as a point of clarification, I placed the `.ebextensions` directory in the project directory, not in the `.elasticbeanstalk` directory. — abogaard, Oct 08 '20 at 14:59
I've removed the reference to elasticbeanstalk, this is because of how I arrange project locally, thanks for pointing this out. — Matthew Buckett, Oct 08 '20 at 15:50
Ok some time it appears that AWS fixed this and with "Amazon Linux 2/3.2.3" it's no longer needed. I don't know exactly when it was fixed. — Matthew Buckett, Jan 12 '21 at 13:26
Whenever it was fixed, they've added a `*` to the ' "file_path": "/var/log/eb-docker/containers/eb-current-app/*stdouterr.log",' . Still would be cool to know where they document fixes like this though. — nelsonjchen, Apr 02 '21 at 21:46

No logs appear on Cloudwatch log group for elastic beanstalk environment

2 Answers2

Linked