0

I'm working on adding log analytics for a simple application. The application is a Windows Service app hosted using NSSM and logs are going to ELK stack (Elastic Search + Kibana). Unfortunately, the app does not utilize a logging framework. It logs to console, and NSSM parses the stdout and stderr and creates log files. The problem is that the multiline logs are completely messed up. NSSM puts the multi-line with carriage returns into multiple log statements with time stamps, which causes the logs to be very noisy and fragmented on Kibana.

This is an example of the logs:

[2/1/2022 10:15:39 PM] Host initialized (4321ms)
[2/1/2022 10:15:40 PM] Host started (4342ms)
[2/1/2022 10:15:40 PM] Job host started
Listening on http://localhost:7071/
Hit CTRL-C to exit...
[2/1/2022 10:15:57 PM] Executing HTTP request: {
[2/1/2022 10:15:57 PM]   "requestId": "[OMITTED]",
[2/1/2022 10:15:57 PM]   "method": "GET",
[2/1/2022 10:15:57 PM]   "uri": "/"
[2/1/2022 10:15:57 PM] }

As you can see in the above example, we have multi-line log statements that are turned into multiple log statements with timestamps for each line. Any suggestions on how to make the data less fragmented and format the data, so that it appears cleaner on Kibana?

Obviously the below solutions come to my mind:

  1. Use a standard logging framework like Log4Net that allows us to specify log severity and have multiline logs per one log statement and configure our logger to
  2. Use Regex in the filebeat config to fix the multi-line issue. The problem is that I don't have a particular pattern to match on.
  3. Use logstash for data processing?

Any help would be appreciated. Thanks!

Vin Shahrdar
  • 1,151
  • 1
  • 12
  • 30

1 Answers1

1

Filebeat can deal with multiline by itself (https://www.elastic.co/guide/en/beats/filebeat/current/multiline-examples.html)

However, if logs look like this - I really doubt it can do better than NSSM did no clear pattern.

Logstash won't help either - it's also pattern-based.

And I must notice that's really quite unusual pain in the bottom area to have a Windows Service spitting output to "console" only.

So if rebuilding your Win Service with a logging framework IS an option - I strongly suggest to go with it.

Or at least make it start message with something uniformal - like timestamp.

Yuri G
  • 1,206
  • 1
  • 9
  • 13
  • Thanks for your thoughts. I will pitch the logging framework idea. Seems like the codebase already has a nice injectable logger, but it only logs to console. Perhaps I'll have it log to both console and Log4net/file logs. Kind of like a Tee-Object. – Vin Shahrdar Feb 08 '22 at 18:20