I am using EFK stack on Kubernetes and I want FluentD to collect logs only from the applications that are in the default namespace, not the other namespaces.
This is my FluentD config:
apiVersion: v1
kind: ConfigMap
metadata:
name: fluentd-config
namespace: logging-kubernetes
data:
fluent.conf: |
<label @FLUENT_LOG>
<match fluent.**>
@type null
</match>
</label>
<match kubernetes.var.log.containers.**kube-system**.log>
@type null
</match>
<source>
@type tail
path /var/log/containers/*.log
pos_file /var/log/app.log.pos
tag kubernetes.*
read_from_head true
<parse>
@type json
time_format %Y-%m-%dT%H:%M:%S.%NZ
</parse>
</source>
<filter kubernetes.**>
@type kubernetes_metadata
</filter>
<filter kubernetes.**>
@type grep
<and>
<exclude>
key log
pattern (.\[notice]\.*|^[ \\\/\(\)\*\|_]+(?!.*[a-zA-Z0-9]).*$|^\s*$|.*GET*|.*POST*)
</exclude>
<exclude>
key kubernetes.namespace_name
pattern ^(?!^default$).*
</exclude>
</and>
</filter>
<match **>
@type elasticsearch
@log_level info
include_tag_key true
host "#{ENV['FLUENT_ELASTICSEARCH_HOST']}"
port "#{ENV['FLUENT_ELASTICSEARCH_PORT']}"
user "#{ENV['FLUENT_ELASTICSEARCH_USER']}"
password "#{ENV['FLUENT_ELASTICSEARCH_PASSWORD']}"
scheme "#{ENV['FLUENT_ELASTICSEARCH_SCHEME'] || 'http'}"
ssl_verify "#{ENV['FLUENT_ELASTICSEARCH_SSL_VERIFY'] || 'true'}"
reload_connections true
logstash_format true
logstash_prefix logstash
<buffer>
@type file
path /var/log/fluentd-buffers/kubernetes.system.buffer
flush_mode interval
retry_type exponential_backoff
flush_thread_count 2
flush_interval 5s
retry_forever true
retry_max_interval 30
chunk_limit_size 2M
queue_limit_length 32
overflow_action block
</buffer>
</match>
I tried doing it this way but it doesn't work it still collects logs from all the namespaces, I also have tried this way:
<regexp>
key kubernetes.namespace_name
pattern /^default$/
</regexp>
but this doesn't work too.
Can someone help me how can I do this?