0

I'm looking at pulling in a log from a mail system where each transaction is over multiple lines.
Each line will have different parts of the transaction and they can be interwoven with other transaction messages - so multiple threads can be involved.
Each transaction has a unique identifier as first field after the time stamp.

I'm looking for assistance in defining the best way of dealing with this situation. Is this a case of needing one logstash to aggregate then another logstash to perfoem the field extractions - ? Any pointer on best process here gratefully received. In the output I would ideally like to see Time, Unique Identifier Source, Destination, Subject

An example of the logs is below:

2016-10-26 20:00:57 xxxxxx-xxxxxx-x1 <= <> H=smtp.org.com (mailrouter.org.com) [10.10.10.10] P=esmtp S=2597 id=201610260900.u9Q90tsK008439@sendingserver.org.com T="Email Subject"
2016-10-26 20:00:57 xxxxxx-xxxxxx-x1 => /dev/null (bounces@destination.com) <provision@destinationaddress.com> R=user_alias T=**bypassed**
2016-10-26 20:00:57 xxxxxx-xxxxxx-x1 Completed
2016-10-26 20:00:57 xxxxxx-xxxxxx-x4 <= <> H=smtp.com (mailrouter.org.com) [10.10.10.10] P=esmtp S=2620 id=201610260900.u9Q90tii008449@sendingserver.org.com T="other email subject"  
2016-10-26 20:00:58 xxxxxx-xxxxxx-x6  <= <> H=Othermail.org.com (mailrouter.org.com) [10.10.10.10] P=esmtp S=2621 id=201610260900.u9Q90tvc008455@sendingserver.org.com T="another email subject"
2016-10-26 20:00:58 xxxxxx-xxxxxx-x6  => /dev/null (bounces@destination.com) <provision@destinationaddress.com> R=user_alias T=**bypassed**
2016-10-26 20:00:58 xxxxxx-xxxxxx-x6  Completed
2016-10-26 20:00:58 xxxxxx-xxxxxx-x9 DKIM: d=customer2.com.au s=cm c=relaxed/relaxed a=rsa-sha1 i=info@customer2.com.au [verification succeeded]
2016-10-26 20:00:58 xxxxxx-xxxxxx-x9 <= Wanted-yhqbhk1didkkhtklr1r@oddmail.com H=mailraouter4.org.com [10.20.20.40] P=esmtp S=29325 id=cm.2000327259153.yhqbhk.didkkhtklr.r@oddmail.com  T="Yet another email subject"
2016-10-26 20:00:58 xxxxxx-xxxxxx-x9 => diane <diane@client.com> R=mysql_localuser T=mysql_delivery
2016-10-26 20:00:58 xxxxxx-xxxxxx-x9 Completed
2016-10-26 20:00:59 xxxxxx-xxxxxx-x4 => /dev/null (bounces@destination.com) <provision@destinationaddress.com> R=user_alias T=**bypassed**
2016-10-26 20:00:59 xxxxxx-xxxxxx-x4 Completed
baudsp
  • 4,076
  • 1
  • 17
  • 35
Jason
  • 3
  • 2
  • You should perhaps take a look to the (aggregate filter plugin)[https://www.elastic.co/guide/en/logstash/current/plugins-filters-aggregate.html]. Also you might get more help if you question show what you have tried and how it does not work (see [how to ask](http://stackoverflow.com/help/how-to-ask)) – baudsp Oct 28 '16 at 09:33
  • Possible duplicate of [How to make Logstash multiline filter merge lines based on some dynamic field value?](http://stackoverflow.com/questions/32304023/how-to-make-logstash-multiline-filter-merge-lines-based-on-some-dynamic-field-va) – Alcanzar Oct 28 '16 at 14:14

0 Answers0