Why does EKS say my fluent-bit.conf is not valid

Question

I am trying to setup Fluent Bit for Kuberentes on EKS + Fargate. I was able to get logs all going to one general log group on Cloudwatch but now when I add fluent-bit.conf: | to the data: field and try to apply the update to my cluster, I get this error:

for: "fluentbit-config.yaml": admission webhook "0500-amazon-eks-fargate-configmaps-admission.amazonaws.com" denied the request: fluent-bit.conf is not valid. Please only provide output.conf, filters.conf or parsers.conf in the logging configmap

What sticks out the most to me is that the error message is asking me to only provide output, filter or parser configurations.

It matches up with other examples I found online, but it seems like I do not have the fluent-bit.conf file on the cluster that I am updating or something. The tutorials I have followed do not mention installing a file so I am lost as to why I am getting this error.

The

My fluentbit-config.yaml file looks like this

kind: Namespace
apiVersion: v1
metadata:
  name: aws-observability
  labels:
    aws-observability: enabled
---
kind: ConfigMap
apiVersion: v1
metadata:
  name: aws-logging
  namespace: aws-observability
  labels:
    k8s-app: fluent-bit
data:
  fluent-bit.conf: |
    @INCLUDE input-kubernetes.conf
    
  input-kubernetes.conf: |
    [INPUT]
        Name tail
        Parser docker
        Tag logger
        Path /var/log/containers/*logger-server*.log
        
  output.conf: |
    [OUTPUT]
        Name cloudwatch_logs
        Match logger
        region us-east-1
        log_group_name fluent-bit-cloudwatch
        log_stream_prefix from-fluent-bit-
        auto_create_group On

score 4 · Accepted Answer · answered Jun 22 '21 at 15:52

As per docs (at the very bottom of that page and yeah, we're in the process of improving them, not happy with the current state) you have a couple of sections in there that are not allowed in the context of EKS on Fargate logging, more specifically what can go into the ConfigMap. What you want is something along the lines of the following (note: this is from an actual deployment I'm using, slightly adapted):

kind: ConfigMap
apiVersion: v1
metadata:
  name: aws-logging
  namespace: aws-observability
data:
  output.conf: |
     [OUTPUT]
        Name cloudwatch_logs
        Match *
        region eu-west-1
        log_group_name something-fluentbit
        log_stream_prefix fargate-
        auto_create_group On
     [OUTPUT]
        Name  es
        Match *
        Host blahblahblah.eu-west-1.es.amazonaws.com
        Port 443
        Index something
        Type  something_type
        AWS_Auth On
        AWS_Region eu-west-1
        tls   On

With this config, you're streaming logs to both CW and AES, so feel free to drop the second OUTPUT section if not needed. However, you notice that there can not be the other sections that you had there such as input-kubernetes.conf for example.

Okay I see what you were referring to. So you cannot work with a fluent-bit.conf or use the @INCLUDE operator when working with EKS Fargate and Fluent bit. Your answer definitely answered my question Michael, but it does lead me to a new question. Can I create different log groups on CW for different services that are logging? I was intending to tag different logs with [INPUT] and match these tags with different [OUTPUTS]. I can create a new question regarding this if you'd rather talk on a new post. Thanks again for your response. — Frederick Haug, Jun 22 '21 at 16:05
Yes, you can have multiple `OUTPUT` sections for different (as I did, CW and AES) or the same destination, `Match`ing on different things, however, the `INPUT` section is not allowed, that's managed EKS/Fargate-internally, in the so called hide-car (== hidden side car). Hope this makes sense and as I said, working on an update of said docs page to make this clearer. — Michael Hausenblas, Jun 23 '21 at 09:08

Blazej Es · Answer 2 · 2022-03-03T15:13:07.107

I wonder if anyone managed to process the 'log' section with fargate 'hide-car' using parser as per fluentbit conf documentation. Here's a snippet of my aws-logging config map which pushes logs to both outputs but sadly the parsing is never happening.

I would like to avoid using hacky regexes when viewing logs in Opensearch which can be avoided with proper parsing of the 'logs'.

PS. I noticed fluentbit docs refer to so called 'docker' parser but fargate nodes are using containerd as the container runtime which could potentially be a problem?

data:
  filters.conf: |
    [FILTER]
        Name             kubernetes
        Match            kube.*
        Merge_Log           On
        Merge_Log_Key       log_proccessed
        Buffer_Size         0
        Kube_Meta_Cache_TTL 300s
        Parser docker
  flb_log_cw: 'true'
  output.conf: |
    [OUTPUT]
        Name cloudwatch_logs
        Match   *
        region eu-west-1
        log_group_name /aws/eks/bs-277-main/container
        log_stream_prefix log-
    [OUTPUT]
        Name  es
        Match *
        Host  vpc-my-amazing-os-endpoint.eu-west-1.es.amazonaws.com
        Port  443
        Index kubernetes
        Type  doc
        AWS_Auth On
        AWS_Region eu-west-1
        tls   On
  parsers.conf: |
    [PARSER]
        Name         docker
        Format       json
        Time_Key     time
        Time_Format  %Y-%m-%dT%H:%M:%S.%L
        Time_Keep    On

Came across this example of fluentbit config with containerd log parsing but it is based on adding Parser param to [INPUT] section which is ignored in Fargate as it is presumably managed by AWS.

It is very unfortunate that crucial component of observability such as fluentbit has so little documentation on AWS Fargate.

Why does EKS say my fluent-bit.conf is not valid

2 Answers2