0

I'm using Logstash to match Fortinet analyzer logs, and the problem is there are so many pattern without order of the fields.

e.g. one type of message would be:

service=DNS hostname="a.b.net" profile="Dns" action=blocked reqtype=direct url="/" sentbyte=0 rcvdbyte=0 direction=N/A msg="URL belongs to a denied category in policy" method=domain cat=61 catdesc="Phishing" crscore=60 crlevel=high

...and another is:

msg="File is infected." action=blocked service=HTTP sessionid=33137 direction=incoming filename="favicon.ico" quarskip=No-skip virus="MSWord/Agent.DD60!tr" dtype="Virus" ref="http://www.fortinet.com/ve?vn=MSWord%2FAgent.DD60%21tr" virusid=6920465 profile="AV"

As you can see both have msg, action, service and profile but with different order.

Is there anyway to build a pattern to match something like: 

(.*?)=%{DATA:\1?}\s

...while giving the field the name of the first match?

Alan Moore
  • 73,866
  • 12
  • 100
  • 156
eladelad
  • 99
  • 2
  • 10
  • It seems `^(?=.*msg="(?[^"]*)")(?=.*action=(?\S*))(?=.*service=(?\S*))(?=.*profile="(?[^"]*)")` can help. Please check. It requires the presence of all 4 parameters though. If some or all can be missing use `?` after the named captures. – Wiktor Stribiżew Nov 03 '15 at 17:42
  • Is this one big file of records? If individual fields are out of order, how do you know where one record stops and the next begins? Also, what's the capability of the regex engine you're using? Like does it do conditionals? –  Nov 03 '15 at 19:45

1 Answers1

3

Use the kv{} filter which can split it all apart and doesn't care about the order.

Alain Collins
  • 16,268
  • 2
  • 32
  • 55
  • Using kv filter solved everything. Much appreciated, only now I understand kv stands for key value... Thanks – eladelad Nov 04 '15 at 07:03