0

I have created Python Application which is writing 3 types of log files in /var/log/{envName}/{applicationName} directory. One is stats logs, second is exception logs and third is access logs/application logs

Now I am using Fluentbit DaemonSet for log shipping to my central logging tool. I want to add kubernetes metadata like podName, PodId, DockerId, Hostname etc to my all log files. I am trying to use Kubernetes filter to add all above metadata but they are reading metadata from tags which is present in log file name. i.e. Log file name should be {podname}{namespace}{container_name}_{dockerId}.log and it should be exists in /var/log/container path.

Is there any way to add k8s metadata to my custom application logs files which is present in different directory.

fluentbit_configmap.yaml

[INPUT]
        Name              tail
        Tag               services.*
        Path              /var/log/dev/mario/*/*_service.log
        Parser            docker
        #DB                /var/log/flb_kube.db
        Mem_Buf_Limit     512MB
        Skip_Long_Lines   On
        Refresh_Interval  10

  filter-kubernetes.conf: |
    [FILTER]
        Name                kubernetes
        Match               *
        Kube_URL            https://kubernetes.default.svc.cluster.local:443
        Merge_Log           On
        Merge_Log_Key       log
        K8S-Logging.Exclude On
        K8S-Logging.Parser  On

    [FILTER]
        Name            nest
        Match           *
        Operation       lift
        Nested_under    kubernetes
        Add_prefix      kubernetes.

    [FILTER]
        Name    modify
        Match   *
        Copy    kubernetes.namespace_name applicationName
        Copy    kubernetes.container_name subsystemName
        Copy    kubernetes.host computerName
        Remove  kubernetes.container_hash
        Remove  kubernetes.docker_id
        Remove  kubernetes.pod_id

    [FILTER]
        Name            nest
        Match           *
        Operation       nest
        Wildcard        kubernetes.*
        Nest_under      kubernetes
        Remove_prefix   kubernetes.

    [FILTER]
        Name        nest
        Match       *
        Operation   nest
        Wildcard    kubernetes
        Wildcard    log
        Wildcard    time
        Wildcard    stream
        Nest_under  json

I have mounted the volumes and able to read logs lines. But I am not able to add k8s metadata into this. I am getting error like Invalid Pattern for given tag

Please let me know if you need more information regarding my configurations.

Thanks

user3645742
  • 111
  • 1
  • 11

1 Answers1

0

you declared 'docker' as parser but haven't defined it. Add [PARSER] with it's attribute or remove the docker parser line.

Shekhar Ruge
  • 1
  • 2