1

I've several modules deployed in my kubernetes cluster and I use fluentBit to read logs for all modules from /var/log/containers folder and write them to a file on a persistent volume. I'm able to successfully write all log files to a single folder in my output path. Now, I want to write the logs of each module in a separate subfolder in my output path named as deployment/release name of each module. The log file is named in the following format pod-name_namespace-name_deployment-name (i.e, each of the name is separated by an underscore _). How do I extract this information from my log file name and use it in the output path in FluentBit configuration?

My current configuration looks like this:

    [SERVICE]
        Flush 1
        Log_Level info
        HTTP_Server On
        HTTP_Listen 0.0.0.0
        HTTP_Port {{ .Values.metricsPort }}

    [INPUT]
        Name tail
        Path /var/log/containers/*.log
        Tag kube.*
        Mem_Buf_Limit 50MB
        Skip_Long_Lines Off
        Read_from_Head true
        Inotify_Watcher false

    [FILTER]
        Name kubernetes
        Match kube.*
        Merge_Log On
        Keep_Log Off
        Labels Off
        Annotations Off
        K8S-Logging.Parser On
        K8S-Logging.Exclude On

    [OUTPUT]
        Name file
        Match kube.*
        Path /output/path
        Mkdir True

And this makes my tag value of the form kube.var.log.containers.pod-name_namespace-name_chart-name, I want to do something like:

[OUTPUT]
        Name file
        Match kube.*
        Path /output/path/${tag.deployment_name}/${tag.pod-name}.log
        Mkdir True

What is the correct way of doing this? From the documentation it seems like Tag_Regex can be used for doing this but how it can be used is not very clear.

I tried the following configuration:

[INPUT]                                
          Name tail                          
          Path /var/log/containers/*.log     
          Tag_Regex ^(?<pod_name>[^_]+)-[^_]+_(?<namespace_name>[^_]+)_(?<deployment_name>[^.]+)\.log$
          Mem_Buf_Limit 50MB                                                                          
          Skip_Long_Lines Off                                                                         
          Read_from_Head true                                                                         
          Inotify_Watcher false                                                                       
                                                                                                      
      [OUTPUT]                                                                                        
          Name file                                                                                   
          Match *                                                                                     
          Path /logs/precisely/${namespace_name}/${deployment_name}.log                               
          Mkdir True , doe not produce any output

but no folder was created.

0 Answers0