0

I am using EFK stack on Kubernetes and I want FluentD to collect logs only from the applications that are in the default namespace, not the other namespaces.

This is my FluentD config:

apiVersion: v1
kind: ConfigMap
metadata:
  name: fluentd-config
  namespace: logging-kubernetes
data:
  fluent.conf: |
    <label @FLUENT_LOG>
       <match fluent.**>
          @type null
       </match>
    </label>

    <match kubernetes.var.log.containers.**kube-system**.log>
        @type null
    </match>

    <source>
      @type tail
      path /var/log/containers/*.log
      pos_file /var/log/app.log.pos
      tag kubernetes.*
      read_from_head true
      <parse>
        @type json
        time_format %Y-%m-%dT%H:%M:%S.%NZ
      </parse>
    </source>

    <filter kubernetes.**>
        @type kubernetes_metadata
    </filter>

    <filter kubernetes.**>
       @type grep
        <and>
          <exclude>
             key log
             pattern (.\[notice]\.*|^[ \\\/\(\)\*\|_]+(?!.*[a-zA-Z0-9]).*$|^\s*$|.*GET*|.*POST*)
          </exclude>
          <exclude>
             key kubernetes.namespace_name
             pattern ^(?!^default$).*
          </exclude>
        </and>
    </filter>

    <match **>
       @type elasticsearch
       @log_level info
       include_tag_key true
       host "#{ENV['FLUENT_ELASTICSEARCH_HOST']}"
       port "#{ENV['FLUENT_ELASTICSEARCH_PORT']}"
       user "#{ENV['FLUENT_ELASTICSEARCH_USER']}"
       password "#{ENV['FLUENT_ELASTICSEARCH_PASSWORD']}"
       scheme "#{ENV['FLUENT_ELASTICSEARCH_SCHEME'] || 'http'}"
       ssl_verify "#{ENV['FLUENT_ELASTICSEARCH_SSL_VERIFY'] || 'true'}"
       reload_connections true
       logstash_format true
       logstash_prefix logstash
       <buffer>
           @type file
           path /var/log/fluentd-buffers/kubernetes.system.buffer
           flush_mode interval
           retry_type exponential_backoff
           flush_thread_count 2
           flush_interval 5s
           retry_forever true
           retry_max_interval 30
           chunk_limit_size 2M
           queue_limit_length 32
           overflow_action block
       </buffer>
    </match>

I tried doing it this way but it doesn't work it still collects logs from all the namespaces, I also have tried this way:

         <regexp>
            key kubernetes.namespace_name
            pattern /^default$/
         </regexp>

but this doesn't work too.

Can someone help me how can I do this?

Daniela
  • 1
  • 1
  • Looks like your `key` syntax is not correct. See https://docs.fluentd.org/plugin-helper-overview/api-plugin-helper-record_accessor#syntax. Should be `$.kubernetes.namespace_name`. – Azeem Jun 21 '23 at 04:41
  • @Azeem Yes, that was the problem, thank you very much for the help. – Daniela Jun 21 '23 at 07:08
  • Awesome! You're welcome! Closing it as a typo. – Azeem Jun 21 '23 at 08:05

0 Answers0