Fluent Bit part of a string in one of the output field?

Question

I am using AWS Firelens to send logs to Cloudwatch. Firelens sends these below fields by default to fluentbit container along with the log message.

{
    "container_id": "asdfasdfasdfadsf",
    "container_name": "/ecs-microservice-ms-233-microservice-ms-e8e2e9e3b3c9e5e02100",
    "ecs_cluster": "arn:aws:ecs:us-east-1:1234:cluster/app-microservices",
    "ecs_task_arn": "arn:aws:ecs:us-east-1:1234:task/asdf",
    "ecs_task_definition": "microservice-ms:233"
}

I want to do something like this in output section,

[OUTPUT]
    Name cloudwatch
    Match **
    region us-east-1
    log_group_name /ecs/${microservice}
    log_stream_prefix ${TAG}
    auto_create_group true

Where, log group name should be the ecs service name. But I don't have any fields with service name. ecs_task_definition is the closest but it has task revision number.

Is it possible to achieve what I am trying to do?

score 1 · Accepted Answer · answered Jun 01 '20 at 08:58

Fluentbit allows us to use environment variables in the output section. Being said that, the solution to the problem is pass an environment variable to the Firelens container.

In AWS ECS this has to be passed from the task definition section, inside the firelens sidecar container definition,

    {
        "name": "firelens_log_router",
        "image": "<account_id>.dkr.ecr.us-east-1.amazonaws.com/accel-aws-for-fluent-bit:latest",
        "logConfiguration": {
            "logDriver": "awslogs",
            "options": {
                "awslogs-group": "firelens-container",
                "awslogs-region": "us-east-1",
                "awslogs-create-group": "true",
                "awslogs-stream-prefix": "firelens"
            }
        },
        "memory": 100,
        "memoryReservation": 50,
        "firelensConfiguration": {
            "type": "fluentbit",
            "options": {
                "config-file-type": "file",
                "config-file-value": "/fluent-bit/etc/fluent-bit-accel.conf"
            }
        },
        "environment": [
            {
                "name": "MICROSERVICE_NAME",
                "value": "${microservice_name}"
            }
        ]
    }

Then in the OUTPUT section of Fluentbit configuration file,

[OUTPUT]
    Name cloudwatch
    Match **
    region us-east-1
    log_group_name /ecs/${MICROSERVICE_NAME}
    log_stream_prefix fluentbit.
    auto_create_group true

Fluent Bit part of a string in one of the output field?

1 Answers1