0

I need a help with task for parsing and combining exim_mainlog for ELK.

Issue is next:

My logstash multiline plugin won't collect lines of log file with unique message id into one event. When I try to send 4 strings in right order it works good. Order like this:

2017-04-10 00:00:30 1cxKsn-0001GB-2t CTAS=IN RefID= ( ISpam= IFlags=v=2.2 cv=Op4/823t c=1 sm=1 tr=0 a=6HVp5djceeYjte4jJb6Ryw==:17 a=AzvcPWV-tVgA:10 a=uHJYF-HtSykr7tHsIToA:9 a=CTTii-5M3Z-LMe4tr8cA:9 a=QEXdDO2ut3YA:10 a=pyshpDcKeHPZtuIe0Z8A:9 )
2017-04-10 00:00:30 1cxKsn-0001GB-2t <= email@domain.com H=m37s3-2-28db.ispgateway.com [176.221.47.15] P=smtp S=2567 id=201704092200.v39M0Qxr016654@m37s3-2-28db.ispgateway.com
2017-04-10 00:00:30 1cxKsn-0001GB-2t => info@domainx.com R=internal_gw T=remote_smtp H=192.168.1.11 [192.168.1.11] C="250 OK id=1cxKso-0002iK-Q7"
2017-04-10 00:00:30 1cxKsn-0001GB-2t Completed

If otder is right - everything works good. But when between lines of the same event there are inserted other trash info it breaks down.

Actual logs are look like this:

2017-04-10 00:00:30 1cxKsn-0001GB-2t CTAS=IN RefID= ( ISpam= IFlags=v=2.2 cv=Op4/823t c=1 sm=1 tr=0 a=6HVp5djceeYjte4jJb6Ryw==:17 a=AzvcPWV-tVgA:10 a=uHJYF-HtSykr7tHsIToA:9 a=CTTii-5M3Z-LMe4tr8cA:9 a=QEXdDO2ut3YA:10 a=pyshpDcKeHPZtuIe0Z8A:9 )
2017-04-10 00:00:30 1cxKsn-0001GB-2t <= email@domain.com H=m37s3-2-28db.ispgateway.com [176.221.47.15] P=smtp S=2567 id=201704092200.v39M0Qxr016654@m37s3-2-28db.ispgateway.com
2017-04-10 00:00:30 1cxKsn-0001GB-2t => info@domainx.com R=internal_gw T=remote_smtp H=192.168.1.11 [192.168.1.11] C="250 OK id=1cxKso-0002iK-Q7"
2017-04-10 00:00:30 1cxKsn-0001GB-2t Completed
2017-04-10 00:00:30 fixed_login authenticator failed for (faYNpaLtF) [192.168.24.24]: 535 Incorrect authentication data
2017-04-10 00:00:30 fixed_login authenticator failed for (lkLmh6Lk) [192.168.24.24]: 535 Incorrect authentication data
2017-04-10 00:00:30 fixed_login authenticator failed for (dLKdHZ) [192.168.24.24]: 535 Incorrect authentication data
2017-04-10 00:00:30 H=mx4.rissoidupgrades.com [79.137.110.132] F=<rtcjrc-cmok892@rissoidupgrades.com> rejected RCPT <qfuohabte_p145@verim.de>: ICIR16 - unknown user
2017-04-10 00:00:30 unexpected disconnection while reading SMTP command from ([111.111.111.111]) [117.241.112.188] (error: Connection reset by peer)
2017-04-10 00:00:30 1cxKso-0001GQ-1R CTAS=IN RefID= ( ISpam=Confirmed IFlags=v=2.2 cv=Op4/823t c=1 sm=1 tr=0 a=LMNu0MzFDzFZvX0DaJwgIA==:17 a=AwJkFeBFn10A:10 a=AzvcPWV-tVgA:10 a=HFQ-CQzmNWWYERzML24A:9 )
2017-04-10 00:00:31 1cxKso-0001GQ-1R <= kd123456@abcdrfg.managed.com H=abcdrfg.managed.com [62.138.219.130] P=esmtp S=671 id=20170409220030.5BCED80909@ma60655.psmanaged.com
2017-04-10 00:00:30 fixed_login authenticator failed for (faYNpaLtF) [192.168.24.24]: 535 Incorrect authentication data
2017-04-10 00:00:30 fixed_login authenticator failed for (lkLmh6Lk) [192.168.24.24]: 535 Incorrect authentication data
2017-04-10 00:00:30 fixed_login authenticator failed for (dLKdHZ) [192.168.24.24]: 535 Incorrect authentication data
2017-04-10 00:00:30 H=mx4.rissoidupgrades.com [79.137.110.132] F=<sdfsdg-sdfsd34@downgrades.com> rejected RCPT <sdfsdf_dsf343@varum.com>: ICIR16 - unknown user
2017-04-10 00:00:30 unexpected disconnection while reading SMTP command from ([117.241.112.188]) [117.241.112.188] (error: Connection reset by peer)
2017-04-10 00:00:31 1cxKso-0001GQ-1R => sarah@tele.com R=internal_gw T=remote_smtp H=192.168.1.11 [192.168.1.11] C="250 OK id=1cxKsp-0002iR-QJ"
2017-04-10 00:00:31 1cxKso-0001GQ-1R Completed

At the end I want to have two events with id 1cxKsn-0001GB-2t and 1cxKso-0001GQ-1R at my Kibana.

Here is my patterns:

EXIM_MSGID [0-9A-Za-z]{6}-[0-9A-Za-z]{6}-[0-9A-Za-z]{2}
EXIM_FLAGS (<=|[-=>*]>|[*]{2}|==)
EXIM_DATE %{YEAR:exim_year}-%{MONTHNUM:exim_month}-%{MONTHDAY:exim_day} %{TIME:exim_time}
EXIM_DATE_EMPTY %{YEAR}-%{MONTHNUM}-%{MONTHDAY} %{TIME}
EXIM_PID \[%{POSINT}\]
EXIM_QT ((\d+y)?(\d+w)?(\d+d)?(\d+h)?(\d+m)?(\d+s)?)
EXIM_EXCLUDE_TERMS (Message is frozen|(Start|End) queue run| Warning: | retry time not reached | no (IP address|host name) found for (IP address|host) | unexpected disconnection while reading SMTP command | no immediate delivery: |another process is handling this message)
EXIM_REMOTE_HOST (H=(%{NOTSPACE:remote_hostname} )?(\(%{NOTSPACE:remote_heloname}\) )?\[%{IP:remote_host}\])
EXIM_INTERFACE (I=\[%{IP:exim_interface}\](:%{NUMBER:exim_interface_port}))
EXIM_PROTOCOL (P=%{NOTSPACE:protocol})
EXIM_MSG_SIZE (S=%{NUMBER:exim_msg_size})
EXIM_HEADER_ID (id=%{NOTSPACE:exim_header_id})
EXIM_SUBJECT (T=%{QS:exim_subject})
NUM_EMAIL (%{HOSTNAME}\@%{HOSTNAME})
EXIM_RECEIVER (=>\s*%{EMAILADDRESS:receiver}(\s*<%{EMAILADDRESS:envelope_sndr}>)?|=>\s*%{NUM_EMAIL:receiver}(\s*<%{EMAILADDRESS:envelope_sndr}>)?)
EXIM_ROUTER (R=%{WORD:router})
EXIM_TRANSPORT (T=%{WORD:transport})
EXIM_REMOTE_SMTP_CONFIRM (C="%{GREEDYDATA:smtp_remote_response}")



EXIM_SPAM %{EXIM_DATE_EMPTY} %{EXIM_MSGID} CTAS=%{WORD:exim_spam_dest} RefID=(%{WORD:exim_refid})? \( (I|O)Spam=(%{WORD:exim_spam})? ((I|O)Virus=%{WORD:exim_virus} )?(I|O)Flags=(%{GREEDYDATA:exim_spam_flags})? cv=%{GREEDYDATA:exim_spam_other} \)

EXIM_LEFT %{EXIM_DATE_EMPTY} %{EXIM_MSGID} %{EXIM_FLAGS:exim_flags} %{GREEDYDATA:exim_email} (%{EXIM_REMOTE_HOST})? %{EXIM_PROTOCOL} (?:X=%{GREEDYDATA:exim_auth_details})?(?:A=%{GREEDYDATA:exim_authenticator})?(?:%{EXIM_MSG_SIZE:exim_mes_size})? (?:id=%{NUM_EMAIL:exim_uid})?

EXIM_RIGHT %{EXIM_DATE_EMPTY} %{EXIM_MSGID} %{EXIM_RECEIVER} %{EXIM_ROUTER} %{EXIM_TRANSPORT} %{EXIM_REMOTE_HOST} %{EXIM_REMOTE_SMTP_CONFIRM}

EXIM_SPAM_CHECK_ST %{EXIM_DATE} %{EXIM_MSGID:exim_msgid} Completed

Here is my filter.conf:

filter {
  if [type] == "exim" {
      multiline {
        patterns_dir   => "/etc/logstash/patterns.d"
        pattern => "%{EXIM_DATE} %{EXIM_MSGID:msgid}"
        what => "previous"
      }
      grok {
        patterns_dir   => "/etc/logstash/patterns.d"
        break_on_match => false
        match          => [ "message", "^%{EXIM_SPAM}" ]
       }
       grok {
         patterns_dir   => "/etc/logstash/patterns.d"
         break_on_match => false
         match          => [ "message", "^%{EXIM_LEFT}" ]
      }

      grok {
        patterns_dir   => "/etc/logstash/patterns.d"
        break_on_match => false
        match          => [ "message", "^%{EXIM_RIGHT}" ]
     }
     grok {
        patterns_dir   => "/etc/logstash/patterns.d"
        break_on_match => false
        match          => [ "message", "^%{EXIM_SPAM_CHECK_ST}" ]
     }
   }
}
baudsp
  • 4,076
  • 1
  • 17
  • 35
Bohdan Kostiuk
  • 1
  • 2

1 Answers1

0

For cumulative collecting information about one event in multiline log you have some ways:

  • using "Aggregate filter plugin"
  • using "ElasticSearch filter plugin"

In first you need to collect all event info into one MessageId. But this can be fraught with a lot of problems - some log lines havn't MessageId, andalso many Exim workers write own lines in mixed order. In the second, and if you are using ElastisSearch to store event info, you can make any additional request for searching previously saved events and update their fileds.

Some example with ES like this https://gist.github.com/greem/6e02b57ff26eaacb01b2

Markevych Alexander
  • 1