Getting custom elastic beanstalk logs into cloudwatch group

Question

Having issues and been unable to create a custom cloudwatch log group from .ebextentions/logs.config

Here are different files I have tried.

1

--- 
files: 
  /opt/elasticbeanstalk/tasks/bundlelogs.d/celery_logs.conf: 
    content: |-
        /var/log/celery_beat.stdout.log
        /var/log/celery_flower.stdout.log
        /var/log/celery_worker.stdout.log
        /var/log/faust_worker.stdout.log
    group: root
    mode: "000755"
    owner: root

2

--- 
files: 
  "/opt/elasticbeanstalk/config/private/logtasks/bundle/applogs.conf" :
    mode: "000755"
    owner: root
    group: root
    content: |
      /var/log/celery_beat.stdout.log
      /var/log/celery_flower.stdout.log
      /var/log/celery_worker.stdout.log
      /var/log/faust_worker.stdout.log

3

packages:
  yum:
    awslogs: []

files:
  "/etc/awslogs/awscli.conf" :
    mode: "000600"
    owner: root
    group: root
    content: |
      [plugins]
      cwlogs = cwlogs
      [default]
      region = `{"Ref":"AWS::Region"}`

  "/etc/awslogs/config/logs.conf" :
    mode: "000600"
    owner: root
    group: root
    content: |
      [/var/log/celery_beat.stdout.log]
      log_group_name = `{"Fn::Join":["/", ["/aws/elasticbeanstalk", { "Ref":"AWSEBEnvironmentName" }, "/var/log/celery_beat.stdout.log"]]}`
      log_stream_name = {instance_id}
      file = /var/log/celery_beat.stdout.log

      [/var/log/celery_flower.stdout.log]
      log_group_name = `{"Fn::Join":["/", ["/aws/elasticbeanstalk", { "Ref":"AWSEBEnvironmentName" }, "/var/log/celery_flower.stdout.log"]]}`
      log_stream_name = {instance_id}
      file = /var/log/celery_flower.stdout.log

      [/var/log/celery_worker.stdout.log]
      log_group_name = `{"Fn::Join":["/", ["/aws/elasticbeanstalk", { "Ref":"AWSEBEnvironmentName" }, "/var/log/celery_worker.stdout.log"]]}`
      log_stream_name = {instance_id}
      file = /var/log/celery_worker.stdout.log

commands:
  "01":
    command: systemctl enable awslogsd.service
  "02":
    command: systemctl restart awslogsd

The logs are properly showing up in the files:

/var/log/celery_beat.stdout.log
/var/log/celery_flower.stdout.log
/var/log/celery_worker.stdout.log
/var/log/faust_worker.stdout.log

But no log group is being created with no logs being transferred to it.

I've tried 15 or more other similar configurations with no luck.

Did you add the CloudWatchAgentServerPolicy and AmazonEC2RoleforSSM to the aws-elasticbeanstalk-ec2-role ? — littleforest, Aug 23 '22 at 23:40
I am having the same issue, but I only tried adding a `conf` to `/opt/elasticbeanstalk/tasks/taillogs.d/` -- but it looks like you tried other locations with no success so I won't go through the trouble. I assume you are on AL2? — vinhboy, Dec 04 '22 at 18:35

vinhboy · Answer 1 · 2022-12-05T19:56:03.343

Someone left an answer but it got deleted so I'll go ahead and just leave it here for y'all.

I think the reason OP, and I, couldn't get it working is because we went down the wrong path by following this tutorial:

https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/using-features.logging.html

In this tutorial they teach you to use a conf file in /opt/elasticbeanstalk/tasks/, but this is only for exporting the logs into the Elastic Beanstalk web console, NOT into CloudWatch logs.

To export custom logs into CloudWatch logs you have to use this tutorial (which is option #3 in OP's question)

https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/AWSHowTo.cloudwatchlogs.html

It specifically has a section for "Custom log file streaming" -- which is done by using .ebextensions to install a package called awslogs and configuring it yourself. This process is way more involved than simply putting in a conf file but that's how it has to be done.

Be sure to also check the Elastic Beanstalk role to make sure it has permissions to interact with CloudWatch logs.

Getting custom elastic beanstalk logs into cloudwatch group

1 Answers1