How to get JSON-tokenized logs into cloudwatch with fluent bit in EKS 1.22 (containerd)

Question

I would like to get the message of my log entry into AWS with correct json tokenization from CRI application logs when running in AWS EKS (version 1.22)

My application outputs valid json, but the log line is prepended with YYYY-MM-DDTHH:MM:SS.MILLIZ stdout F. This means that a standard JSON parser does not work.

Using a regex parser from http://rubular.com/r/tjUt3Awgg4, the time, stream and logtag from my above example are all correctly identified, but I then cannot tokenize the message JSON. My log content is then in the message key.

[PARSER]
        # http://rubular.com/r/tjUt3Awgg4
        Name cri
        Format regex
        Regex ^(?<time>[^ ]+) (?<stream>stdout|stderr) (?<logtag>[^ ]*) (?<message>.*)$
        Time_Key    time
        Time_Format %Y-%m-%dT%H:%M:%S.%L%z

Is there a way to stack parsers, or a way to remove the first part of the log line before sending it to the parser? I attempted a remove_regex in a filter, but couldn't make that work.

score 1 · Answer 1 · answered Oct 05 '22 at 21:12

We are using EKS with Fargate and this configuration fixes the problem for us.

kind: ConfigMap
apiVersion: v1
metadata:
  name: aws-logging
  namespace: aws-observability
data:
  output.conf: |
    [OUTPUT]
        Name cloudwatch_logs
        Match   *
        region region-code
        log_group_name fluent-bit-cloudwatch
        log_stream_prefix from-fluent-bit-
        auto_create_group true
        log_key log

  parsers.conf: |
    [PARSER]
        Name crio
        Format Regex
        Regex ^(?<time>[^ ]+) (?<stream>stdout|stderr) (?<logtag>P|F) (?<log>.*)$
        Time_Key    time
        Time_Format %Y-%m-%dT%H:%M:%S.%L%z
  
  filters.conf: |
     [FILTER]
        Name parser
        Match *
        Key_name log
        Parser crio

As found at https://docs.aws.amazon.com/eks/latest/userguide/fargate-logging.html

Samuel · Answer 2 · 2023-06-28T16:31:06.370

We use the built-in multiline parser for CRI-O and the Merge_Log parameter in Kubernetes filter and it works well in our case:

[INPUT]
    Name                tail
    Path                /var/log/containers/*.log
    multiline.parser    cri
...

[FILTER]
    Name                kubernetes
    Merge_Log           On
    Merge_Log_Key       log_processed
...

Fluent bit documentation:

How to get JSON-tokenized logs into cloudwatch with fluent bit in EKS 1.22 (containerd)

2 Answers2