fluentd config to parse auth.log

Question

I'm using the following config to have fluentd read the auth.logs and send it to elastic search but i'm faced with an error saying pattern doesn't match and the logs are not pushed to ES.

I'm using the pattern defined in fluentd syslog parser plugin rfc3164-pattern

<source>
  @type tail
  path /var/log/auth.log
  pos_file /var/log/auth.pos
  format /^\<(?<pri>[0-9]+)\>(?<time>[^ ]* {1,2}[^ ]* [^ ]*) (?<host>[^ ]*) (?<ident>[a-zA-Z0-9_\/\.\-]*)(?:\[(?<pid>[0-9]+)\]) *(?<message>.*)$/
  tag authlog
</source>
<match authlog.**>
  @type elasticsearch
  hosts "ESHOST:PORT"
  logstash_format true
  logstash_prefix "server-authlogs"
  include_tag_key true
  flush_interval 5s
  logstash_dateformat %Y.%m.%d
  time_precision 3
</match>

Output Error:

2019-04-16 08:00:50 +0000 [warn]: #0 pattern not match: "Apr 16 08:00:50 hostname-1415 sshd[15134]: pam_unix(sshd:session): session opened for user ubuntu by (uid=0)" 2019-04-16 08:00:50 +0000 [warn]: #0 pattern not match: "Apr 16 08:00:50 hostname-1415 systemd-logind[1138]: New session 10 of user ubuntu."

This is a regex issue, not fluent. Try to troubleshoot with https://fluentular.herokuapp.com/. Note that your error sample has 2 errors in it. — Max Lobur, Apr 16 '19 at 11:51
I did and found a regex to match the auth.log. Thanks for direction. — nevosial, Apr 16 '19 at 19:03

score 0 · Answer 1 · answered Apr 16 '19 at 19:09

For those who are looking for something similar, here is the config that works well.

<source> 
 type tail 
 path /var/log/foo/auth.log 
 pos_file /var/log/auth.pos 
 tag authlog
 format /^(?<time>[^ ]* {1,2}[^ ]* [^ ]*) (?<host>[^ ]*) (?<ident>[a-zA-Z0-9_\/\.\-]*)(?:\[(?<pid>[0-9]+)\])?(?:[^\:]*\:)? *(?<message>.*)$/ 
</source>
<match authlog.**>
 @type elasticsearch
 hosts "ESHOST:PORT"
 logstash_format true
 logstash_prefix "server-authlogs"
 include_tag_key true
 flush_interval 5s
 logstash_dateformat %Y.%m.%d
 time_precision 3
</match>

For a auth.log pattern of:

Apr 16 18:02:02 host-1415 sshd[11111]: Accepted password for ubuntu from 111.11.111.11 port 11111 ssh2

score 0 · Answer 2 · answered Apr 17 '19 at 00:36

How about using parser_syslog? If /var/log/auth.log has syslog format(RFC3164).

<source> 
  @type tail 
  path /var/log/auth.log 
  pos_file /var/log/auth.pos 
  tag authlog
  <parse>
    @type syslog
    message_format rfc3164
    with_priority false
  </parse>
</source>

fluentd config to parse auth.log

2 Answers2