0

Can someone please help me with this question ?

Per our usecase, I have changed the logging driver to "fluentd" in /etc/docker/daemon.json So all pods that gets deployed on the kubernetes nodes will directly forward the log events to the fluentd instace.

we don't have any ElasticSearch, rather we would like to have the received log events be written to the file system, in this particular way.

/data/<namespace>/<service name>/<last 5 characters of POD name>/application.log

and application.log, shall have the log event as is. (timestamp generated by the app and logevent)

I am trying to tweak this below config

[SERVICE]
    Flush        5
    Daemon       Off
    Log_Level    debug

[INPUT]
    Name   forward
    Listen 0.0.0.0
    Port   24224

[FILTER]
  Match               **
  Name                kubernetes
  Merge_Log           On
  K8S-Logging.Parser  On
  Merge_Log        On
  Merge_Log_Key    log_processed
  Annotations      On
  Labels           On

[OUTPUT]
    Name  file
    Path ??
Mr.KoopaKiller
  • 3,665
  • 10
  • 21
Venu S
  • 3,251
  • 1
  • 9
  • 25
  • 1
    Why not just let Docker write to a file, which is already the default? Also this will break `kubectl logs`. – coderanger Apr 16 '20 at 02:17
  • fluent-bit supports the default log to file setup out of the box - https://docs.fluentbit.io/manual/installation/kubernetes – Matt Apr 16 '20 at 02:43
  • @Matt, as per the documentation , Input has to be either tail or systemd. And I prefer using forward https://docs.fluentbit.io/manual/pipeline/filters/kubernetes – Venu S Apr 16 '20 at 04:10
  • @coderanger, If I just let docker write to a file, Its won't be collecting/grouping the logs by the service (and it will be hard to identify the log pertaining to a service, possible but not always easy) and the reason why I want to do this is to have a meaningful and organized logs – Venu S Apr 16 '20 at 04:14
  • I would still probably use a more conventional setup. Use bit DS to tail the normal files -> forward -> fluentd deployment -> file output on an NFS volume or something. This is not going to be great though, why not run a 1-node ElasticSearch or something? Uses more RAM but you can tune it down pretty hard. – coderanger Apr 16 '20 at 05:47
  • @VenuS The symlink names kubernetes creates to the container runtime logs contain the pod name and namespace. When fluent bit tails these, they become the tag the k8s filter then uses to lookup additional data from the API. By default you would lose that via fluentd, are you adding those back in? – Matt Apr 16 '20 at 13:19

0 Answers0