0

We have a td_agent.conf file with the following tag:

#this filter is used for C API which remove "[stdout]" from log
#if CLOG Unified Logging C API won't be used, this filter can be removed
<filter k.**.log>
@type parser
format /^(\[stdout\])*(?<log>.+)$/
key_name log
suppress_parse_error_log true
</filter>

and the following sample log line:

{"host":"omer","level":"TRACE","log":{"classname":"Manager:452","message":"^~\"DD\"-^ TRACE Added context","stacktrace":"","threadname":"Processing-ThreadPool-2"},"process":"Context","service":"","time":"2020-11-04T13:37:12.979Z","timezone":"Kolkata","type":"log"}

When having the above logic in Fluentd, we get the log outputted, with the log: {} emitted, that means not having the info that we want in the elastic db. When removing the tag, it all works fine. Can anyone explain why this is needed?

The start of the td-agent is:

<source>
@type tail
path /var/log/containers/*s*.log
pos_file /var/log/td-agent/containers.json.access.pos
tag k.*
#read_from_head true
<parse>
    @type regexp
    expression /(^(?<header>[^\{]+)?(?<message>\{.+\})$)|(^(?<log>[^\{].+))/
</parse>
</source>


<filter k.var.log.containers.**.log>
  @type parser
  key_name message
  format json
  #time_parse false
  time_key time
  time_format %iso8601
  keep_time_key true
</filter>


#this filter is used for C API which remove "[stdout]" from log
#if CLOG Unified Logging C API won't be used, this filter can be removed
<filter k.**.log>
@type parser
format /^(\[stdout\])*(?<log>.+)$/
key_name log
suppress_parse_error_log true
</filter>
Omer
  • 15
  • 4
  • Which version are you using? – Azeem Nov 10 '20 at 13:08
  • fluentd' version '1.9.2' – Omer Nov 10 '20 at 13:47
  • Right. Are you referring to this `k.**.log` tag in the last part? – Azeem Nov 10 '20 at 14:38
  • Yes, when the log enters the td-agent, it is passed correctly, and after this tag that I wrote, the log is now a flat json, without the : "log":{"classname":"M......."} anymore – Omer Nov 10 '20 at 14:58
  • You need to show your full config especially the input. Please update your question with that as well. – Azeem Nov 11 '20 at 03:14
  • @Azeem I've added the configuration up to the tag. To clarify - after the source and the first filter, the log is correct with the log.classname, log.message... present, and after the C api element, the nested log object is emitted, – Omer Nov 11 '20 at 07:36
  • Isn't `kubernetes.var.log.containers.**.log` different from what you posted earlier i.e. `k.**.log`? I'd suggest to use `stdout` plugin locally and find out if `log` contains any data? After parsing, you get three keys i.e. header, message, and log, right? Your regular expression is not working, please check here: https://rubular.com/r/Fw9dM5JNOECBj8. I believe you need to revise that according to your use-case. – Azeem Nov 11 '20 at 14:25
  • BTW, the log that posted is a valid JSON. Why are you not using `json` parser instead of `regexp`? Wouldn't that simplify it for you? – Azeem Nov 11 '20 at 14:25

0 Answers0