I've several modules deployed in my kubernetes cluster and I use fluentBit to read logs for all modules from /var/log/containers
folder and write them to a file on a persistent volume. I'm able to successfully write all log files to a single folder in my output path. Now, I want to write the logs of each module in a separate subfolder in my output path named as deployment/release name of each module. The log file is named in the following format pod-name_namespace-name_deployment-name
(i.e, each of the name is separated by an underscore _). How do I extract this information from my log file name and use it in the output path in FluentBit configuration?
My current configuration looks like this:
[SERVICE]
Flush 1
Log_Level info
HTTP_Server On
HTTP_Listen 0.0.0.0
HTTP_Port {{ .Values.metricsPort }}
[INPUT]
Name tail
Path /var/log/containers/*.log
Tag kube.*
Mem_Buf_Limit 50MB
Skip_Long_Lines Off
Read_from_Head true
Inotify_Watcher false
[FILTER]
Name kubernetes
Match kube.*
Merge_Log On
Keep_Log Off
Labels Off
Annotations Off
K8S-Logging.Parser On
K8S-Logging.Exclude On
[OUTPUT]
Name file
Match kube.*
Path /output/path
Mkdir True
And this makes my tag value of the form kube.var.log.containers.pod-name_namespace-name_chart-name
, I want to do something like:
[OUTPUT]
Name file
Match kube.*
Path /output/path/${tag.deployment_name}/${tag.pod-name}.log
Mkdir True
What is the correct way of doing this? From the documentation it seems like Tag_Regex can be used for doing this but how it can be used is not very clear.
I tried the following configuration:
[INPUT]
Name tail
Path /var/log/containers/*.log
Tag_Regex ^(?<pod_name>[^_]+)-[^_]+_(?<namespace_name>[^_]+)_(?<deployment_name>[^.]+)\.log$
Mem_Buf_Limit 50MB
Skip_Long_Lines Off
Read_from_Head true
Inotify_Watcher false
[OUTPUT]
Name file
Match *
Path /logs/precisely/${namespace_name}/${deployment_name}.log
Mkdir True , doe not produce any output
but no folder was created.