0

I have very complex log messages, that I want to reduce to the most important fields in order to save quota.

The log messages are multiline and there is a lot of redundant information in them. A solution is to script something to reformat these logs into oneliners before feeding them to the splunk forwarders, but I would prefer to use an already existing solution. I thought about using something like logstash or fluentd, but they don't seem to really match my usecase.

Does anybody have experience with log parsing for Splunk or ELK and knows of a good utility do this?

Thanks & Best Regards!

gspoosi
  • 131
  • 4
  • Without know your exact use case, it seems to me like Logstash is the perfect solution here. How do they not fit? Where are you logs coming from; TCP socket, file, something else? – GregL May 16 '19 at 23:06
  • For this source the files are already on a disk - synced every 5min via rsync. The amount is quite high - about 800GB per day and there absolutely cannot be any gaps. – gspoosi May 17 '19 at 05:44
  • That's a lot of logs, but Logstash can scale up quite well if you have the resources (hardware/VMs, memory, CPU, disk) to support it, so I'm not sure why it wouldn't work. – GregL May 17 '19 at 17:05

1 Answers1

2

This is common when dealing with Windows event logs. Many address the problem by using SEDCMD in transforms.conf to edit the events.

Another solution is to pre-process the events with Cribl (https://www.cribl.io/).

RichG
  • 161
  • 4
  • Hi! Thanks for the inputs. I knew I cannot be the only one with this problem. I will check out if these are the right solutions for me. – gspoosi May 17 '19 at 05:47