Fluentd conf to filter on key_name of log message

Question

I am trying to send the stdout logs of my application running in k8s pods to a remote syslog server. I have the fluentd container running as a sidecar to my main application container and it works to send the log to the remote syslog server. But without any formatting etc, the logs that get sent to the remote syslog sever is in the exact way that it is written in the /var/log/container/my_application*.log file. I just want the value of the "log" key as a raw string to be given as the output from fluentd and sent to the remote syslog server.

Logs written in the /var/log/container/my_application*.log file

{"log":"I am application\n","stream":"stdout","time":"2023-04-29T15:11:50.728436003Z"}

I want to send just the "log" key to the remote syslog sever, which actually contains the logs written by the application.

I have the below fluentd config which I'm passing via a k8s config-map to the application pod:

<source>
  @type tail
  path "/var/log/containers/my_application*default*demo*.log"
  pos_file "/var/log/my_app.log.pos"
  read_from_head true
  tag "app-development"
  <parse>
    @type none
  </parse>
</source>
<filter app-development>
  @type parser
  key_name log
  <parse>
    @type none
  </parse> 
</filter>
<match app-development>
  @type remote_syslog
  @id out_kube_remote_syslog
  host "#{ENV['SYSLOG_HOST']}"
  port "#{ENV['SYSLOG_PORT']}"
  severity debug
  program app-development_p
  hostname ${kubernetes_host}
  facility daemon

  protocol "#{ENV['SYSLOG_PROTOCOL'] || 'tcp'}"
  tls "#{ENV['SYSLOG_TLS'] || 'false'}"
  ca_file "#{ENV['SYSLOG_CA_FILE'] || ''}"
  verify_mode "#{ENV['SYSLOG_VERIFY_MODE'] || ''}"
  packet_size 65535

  <buffer kubernetes_host>
  </buffer>
  
  <system>
    file_permission 666
  </system>
</match>

For some reason this doesn't work. I took a look at this documentation to add the filter section: https://docs.fluentd.org/filter/parser#key_name I get the below error from the fluentd container:

 #0 dump an error event: error_class=ArgumentError error="log does not exist" location=nil tag="app-development" time=2023-04-29 15:23:03.221790676 +0000 record={"message"=>"{\"log\":\"I am application\\n\",\"stream\":\"stdout\",\"time\":\"2023-04-29T15:23:02.448835229Z\"}"}

What am I doing wrong in the fluentd configuration?

The logs are in JSON format but the `parse` section is configured as `none`. Why is that? — Azeem, Apr 29 '23 at 17:41
@Azeem You might have a point there. That was a typo from earlier. I do have the solution now after looking through all fluentd questions on stackoverflow. — user1452759, Apr 29 '23 at 17:53

score 1 · Answer 1 · answered Apr 29 '23 at 18:00

I finally think I have a solution for this after scanning through all fluentd based questions on stackover flow. This was what finally helped me get what I was looking for

<source>
  @type tail
  path "/var/log/containers/my_application*default*demo*.log"
  pos_file "/var/log/my_app.log.pos"
  read_from_head true
  tag "app-development"
  <parse>
    @type json
  </parse>
</source>
<match app-development>
  @type remote_syslog
  @id out_kube_remote_syslog
  .
  .
  .
  .
  <buffer kubernetes_host>
  </buffer>
  #Added the below format section 
  <format>
    @type single_value
    message_key log
  </format>
  <system>
    file_permission 666
  </system>
</match>

The format section above finally gives me the below output from the "log" key

I am application

Fluentd conf to filter on key_name of log message

1 Answers1