1

With the config below:

[INPUT]
    Name tail
    Path /var/log/containers/*.log
    multiline.parser docker, cri
    Tag kube.*
    Mem_Buf_Limit 5MB
    Skip_Long_Lines On

[FILTER]
    Name kubernetes
    Match kube.*
    Merge_Log On
    Keep_Log Off
    K8S-Logging.Parser On
    K8S-Logging.Exclude On

[OUTPUT]
    Name kafka
    Match kube.*
    Brokers <some-broker>
    Topics <some-topic>

FluentBit sends container logs to Kafka with the message value below:

{
   "@timestamp":1670628155.720578,
   "log":"<some-container-log>",
   "stream":"stdout",
   "time":"2022-12-09T23:22:35.720578536Z",
   "kubernetes":{
      "pod_name":"<...>",
      "namespace_name":"<...>",
      "pod_id":"<...>",
      "annotations":{ <...> },
      "host":"<...>",
      "container_name":"<...>",
      "docker_id":"<...>",
      "container_hash":"<...>",
      "container_image":"<...>"
   }
}

Now I want this message to have key. When added below line to Kafka output config:

Message_Key_Field stream

I see that all messages have key stdout, which is the value for stream key in the message value. However, when I want to use a field from "kubernetes" in the JSON, say "pod_name", none of below worked and messages ended up with empty key:

- Message_Key_Field kubernetes.pod_id
- Message_Key_Field kubernetes['pod_id']
- Message_Key_Field $kubernetes['pod_id']

What is the correct syntax to refer to an inner JSON value of the record from output plugin?

kafka-user122
  • 11
  • 3

1 Answers1

0

It should be your third option. Isn't that working for you? What does your full config look like?

José Lecaros Cisterna
  • 1,023
  • 12
  • 15