Prevent _grokparsefailure if one of the multipe groks match

Question

I have this grok:

grok {
        patterns_dir => "/etc/logstash/patterns/"
        break_on_match => false
        keep_empty_captures => true

        match => [ 
            "message", "(%{EXIM_DATE:exim_date} )(%{EXIM_PID:exim_pid} )(%{EXIM_MSGID:exim_msg_id} )(%{EXIM_FLAGS:exim_flags} )(%{GREEDYDATA})",
            "message", "(%{EXIM_MSGID} )(<= )(%{NOTSPACE:env_sender} )(%{EXIM_REMOTE_HOST} )?(%{EXIM_INTERFACE} )?(%{EXIM_PROTOCOL} )?(X=%{NOTSPACE:tls_info} )?(%{EXIM_MSG_SIZE} )?(%{EXIM_HEADER_ID} )?(%{EXIM_SUBJECT})",
            "message", "(%{EXIM_MSGID} )([=-]> )(%{NOTSPACE:env_rcpt} )(<%{NOTSPACE:env_rcpt_outer}> )?(R=%{NOTSPACE:exim_router} )(T=%{NOTSPACE:exim_transport} )(%{EXIM_REMOTE_HOST} )(X=%{NOTSPACE:tls_info} )?(QT=%{EXIM_QT:exim_qt})",
            "message", "(%{EXIM_DATE:exim_date} )(%{EXIM_PID:exim_pid} )(%{EXIM_MSGID:exim_msg_id} )(Completed )(QT=%{EXIM_QT:exim_qt})",
            "message", "(%{EXIM_DATE:exim_date} )(%{EXIM_PID:exim_pid} )(%{EXIM_MSGID:exim_msg_id} )?(%{EXIM_REMOTE_HOST} )?(%EXIM_INTERFACE} )?(F=<%{NOTSPACE:env_sender}> )?(.+(rejected after DATA|rejected \(but fed to sa-learn\)|rejected [A-Z]+ (or [A-Z]+ %{NOTSPACE}?|<%{NOTSPACE:env_rcpt}>)?): (?<exim_rej_reason>.+))"
        ]
      }

If I test the grok patterns individually everything works as expected, but in production with multiple matches they do not. The result is OK, I got everything parsed, but I got every time a _grokparsefailure tag, also if one of the 5 is a match. How do I prevent this?

Tag removal is not what I want because if there is no match the tag should be added so I can drop the message.

score 1 · Answer 1 · answered Feb 28 '20 at 14:29

The reason, that you get a failure is that you have set the break_on_match, which test every entry in your match. This results in one of your patterns not matching and setting the _grokparsefailure tag.

From the looks of it your patterns are all exclusive to one another so you wouldn't need to set the break_on_match and still retain the functionality.

Prevent _grokparsefailure if one of the multipe groks match

1 Answers1