Kubernetes pod name/service name as index name in Kibana using fluentd

Question

Currently we have many services running on k8s and sending logs with fluent-bit to elastic using fluentd.

In fluentd we have hard coded logstash_prefix xxx-logstash, so all logs are created with the same index. Now we want to send data to elastic with respect to podname/service name.

From the json document of logs in kibana, we see there is a key PodName, but how to use this in fluentd.conf? We are using helm for elastic stack deployment.

fluentd.conf

#see more ddetails in https://github.com/uken/fluent-plugin-elasticsearch
apiVersion: v1
kind: ConfigMap
metadata:
  name: elasticsearch-output
data:
  fluentd.conf: |
  
    #configure the logging level to error   
    <system>
      log_level error
    </system>

    # Ignore fluentd own events
    <label @FLUENT_LOG>
      <match fluent.**>
        @type null
      </match>
    </label>  

    # TCP input to receive logs from the forwarders
    <source>
      @type forward
      bind 0.0.0.0
      port 24224
    </source>

    # HTTP input for the liveness and readiness probes
    <source>
      @type http
      bind 0.0.0.0
      port 9880
    </source>

    # Throw the healthcheck to the standard output instead of forwarding it
    <match fluentd.healthcheck>
      @type null
    </match>

    # Send the logs to the standard output
    <match **>
      @type             elasticsearch
      include_tag_key   true
      host              "{{ .Release.Name }}-es-http"
      port              "9200"
      user              "elastic"
      password          "{{ (.Values.env.secret.password | b64dec) | indent 4 | trim }}"
      logstash_format   true
      scheme            https
      ssl_verify        false
      logstash_prefix   xxx-logstash
      logstash_prefix_separator -
      logstash_dateformat %Y.%m.%d

      <buffer>
        @type file
        path /opt/bitnami/fluentd/logs/buffers/logs.buffer
        flush_thread_count 2
        flush_interval 5s
      </buffer>
    </match>

** Sample log document from Kibana**

{
  "_index": "xxx-logstash-2022.08.19",
  "_type": "_doc",
  "_id": "N34ntYIBvWtHvFBZmz-L",
  "_version": 1,
  "_score": 1,
  "_ignored": [
    "message.keyword"
  ],
  "_source": {
    "FileName": "/app/logs/app.log",
    "@timestamp": "2022-08-19T08:10:46.854Z",
    "@version": "1",
    "message": "[com.couchbase.endpoint][EndpointConnectionFailedEvent][1485us] Connect attempt 16569 failed because of : finishConnect(..) failed: Connection refused: xxx-couchbase-cluster.couchbase/10.244.27.5:8091 - Check server ports and cluster encryption setting. {\"circuitBreaker\":\"DISABLED\",\"coreId\":\"0x94bd86a800000002\",\"remote\":\"xxx-couchbase-cluster.couchbase:8091\",\"type\":\"MANAGER\"}",
    "logger_name": "com.couchbase.endpoint",
    "thread_name": "cb-events",
    "level": "WARN",
    "level_value": 30000,
    "stack_trace": "com.couchbase.client.core.endpoint.BaseEndpoint$2: finishConnect(..) failed: Connection refused: xxx-couchbase-cluster.couchbase/10.244.27.5:8091 - Check server ports and cluster encryption setting.\n",
    "PodName": "product-59b7f4b567-r52vn",
    "Namespace": "designer-dev",
    "tag": "tail.0"
  },
  "fields": {
    "thread_name.keyword": [
      "cb-events"
    ],
    "level": [
      "WARN"
    ],
    "FileName": [
      "/app/logs/app.log"
    ],
    "stack_trace.keyword": [
      "com.couchbase.client.core.endpoint.BaseEndpoint$2: finishConnect(..) failed: Connection refused: xxx-couchbase-cluster.couchbase/10.244.27.5:8091 - Check server ports and cluster encryption setting.\n"
    ],
    "PodName.keyword": [
      "product-59b7f4b567-r52vn"
    ],
    "@version.keyword": [
      "1"
    ],
    "message": [
      "[com.couchbase.endpoint][EndpointConnectionFailedEvent][1485us] Connect attempt 16569 failed because of : finishConnect(..) failed: Connection refused: xxx-couchbase-cluster.couchbase/10.244.27.5:8091 - Check server ports and cluster encryption setting. {\"circuitBreaker\":\"DISABLED\",\"coreId\":\"0x94bd86a800000002\",\"remote\":\"xxx-couchbase-cluster.couchbase:8091\",\"type\":\"MANAGER\"}"
    ],
    "Namespace": [
      "designer-dev"
    ],
    "PodName": [
      "product-59b7f4b567-r52vn"
    ],
    "@timestamp": [
      "2022-08-19T08:10:46.854Z"
    ],
    "level.keyword": [
      "WARN"
    ],
    "thread_name": [
      "cb-events"
    ],
    "level_value": [
      30000
    ],
    "Namespace.keyword": [
      "designer-dev"
    ],
    "@version": [
      "1"
    ],
    "logger_name": [
      "com.couchbase.endpoint"
    ],
    "tag": [
      "tail.0"
    ],
    "stack_trace": [
      "com.couchbase.client.core.endpoint.BaseEndpoint$2: finishConnect(..) failed: Connection refused: xxx-couchbase-cluster.couchbase/10.244.27.5:8091 - Check server ports and cluster encryption setting.\n"
    ],
    "tag.keyword": [
      "tail.0"
    ],
    "FileName.keyword": [
      "/app/logs/app.log"
    ],
    "logger_name.keyword": [
      "com.couchbase.endpoint"
    ]
  },
  "ignored_field_values": {
    "message.keyword": [
      "[com.couchbase.endpoint][EndpointConnectionFailedEvent][1485us] Connect attempt 16569 failed because of : finishConnect(..) failed: Connection refused: xxx-couchbase-cluster.couchbase/10.244.27.5:8091 - Check server ports and cluster encryption setting. {\"circuitBreaker\":\"DISABLED\",\"coreId\":\"0x94bd86a800000002\",\"remote\":\"xxx-couchbase-cluster.couchbase:8091\",\"type\":\"MANAGER\"}"
    ]
  }
}