0

I am trying to write a logstash filter for my Java logs so that I can insert them into my database cleanly.

Below is an example of my log format:

FINE  2016-01-28 22:20:42.614+0000  net.myorg.crypto.CryptoFactory:getInstance:73:v181328
AppName   : MyApp  AssocAppName:
Host      : localhost  127.000.000.001  AssocHost:
Thread    : http-bio-8080-exec-5[23]
SequenceId: -1
Logger    : net.myorg.crypto.CryptoFactory
Message   : ENTRY
---
FINE  2016-01-28 22:20:42.628+0000  net.myorg.crypto.CryptoFactory:getInstance:75:v181328
AppName   : MyApp  AssocAppName:
Host      : localhost  127.000.000.001  AssocHost:
Thread    : http-bio-8080-exec-5[23]
SequenceId: -1
Logger    : net.myorg.crypto.CryptoFactory
Message   : RETURN
---

My logstash-forwarder is pretty simple. It just includes all logs in the directory (they all have the same format as above)

"files": [
    {
        "paths":  [ "/opt/logs/*.log" ],
        "fields": { "type": "javaLogs" }
    }
]

The trouble I'm having is on the logstash side. How can I write a filter in logstash to match this log format?

Using something like this, gets me close:

filter {

      if [type] == "javaLogs" {

          multiline {
              pattern => "^%{TIMESTAMP_ISO8601}"
              negate => true
              what => "previous"
          }
      }
}

But I want to break each line in the log down to its own mapping in logstash. For example, creating fields like AppName, AssocHost, Host, Thread, etc.

I think the answer is using grok.

Community
  • 1
  • 1
cookandy
  • 905
  • 1
  • 8
  • 16

1 Answers1

0

Joining them with multiline (the codec or filter, depending on your needs) is a great first step.

Unfortunately, your pattern says "If the log entry doesn't start with a timestamp, join it with the previous eentry".

Note that none of your log entries start with a timestamp.

Alain Collins
  • 16,268
  • 2
  • 32
  • 55
  • Thanks. Yes, the example I provided doesn't really do anything useful, I agree. Can you help provide a working filter to separate my log items as requested? – cookandy Jan 29 '16 at 00:23
  • My answer shows you how to "read" your multiline pattern. From your sample data, you really want it to read "if the log entry doesn't start with 'FINE', join it with the previous entry'. You'll notice that it is very similar to what you have now and will only require a small change. – Alain Collins Jan 29 '16 at 04:39