0

I have a multiline log written in a file as follows:

INFO   | jvm 1    | main    | 2014/11/06 13:41:30.112 | ERROR [appHTTP50] [appEmployeeAuthenticationProvider] Can't login with username 'username'
INFO   | jvm 1    | main    | 2014/11/06 13:41:30.112 | org.framework.security.authentication.BadCredentialsException: Bad credentials
INFO   | jvm 1    | main    | 2014/11/06 13:41:30.112 |     at de.app.platform.security.CoreAuthenticationProvider.authenticate(CoreAuthenticationProvider.java:133)
INFO   | jvm 1    | main    | 2014/11/06 13:41:30.112 |     at ca.canadiantire.security.appEmployeeAuthenticationProvider.authenticate(appEmployeeAuthenticationProvider.java:39)
INFO   | jvm 1    | main    | 2014/11/06 13:41:30.112 |     at org.framework.security.authentication.ProviderManager.authenticate(ProviderManager.java:156)
INFO   | jvm 1    | main    | 2014/11/06 13:41:30.112 |     at org.framework.security.authentication.ProviderManager.authenticate(ProviderManager.java:177)

However, line below is in each line of the trace on the begginning:

INFO | jvm 1 | main | 2014/11/06 13:41:30.112 |

Does anyone know how to leave this line on the beggining near "ERROR" and drop this part of the line in the trace with grok and get full trace as a single message in Logstash? Any other solutions are welcome.

Mikhail Bibik
  • 1
  • 1

1 Answers1

0

I would think gsub{} is the answer. Either have a conditional stanza that would remove the preface from the subsequent lines, e.g.:

if [message] !~ /\| ERROR / {
    mutate {
        gsub => [ "message", "^.* \| ", "" ]
    }
}

which, if it's "greedy" might leave you with a line like this:

org.framework.security.authentication.BadCredentialsException: Bad credentials

which could then be combined with a subsequent multiline{} filter.

Obviously, you'd need to make both regexps generic enough to handle each log level that you're expecting.

Alain Collins
  • 16,268
  • 2
  • 32
  • 55
  • Hi Alain, the question here is to leave only the first line and remove for others in trace as we need to have initial timestamp. Do you have any ideas on how to get it? And another thing is that we need this not only for ERROR, but for WARN and DEBUG as well. So i guess we can replace just with mutate filter plugin (without if statement)? – Mikhail Bibik Nov 09 '15 at 12:56
  • I understood the question, which is why I gave you an answer. I also understood that there would be more than one level, which is why I mentioned that in my answer. No, you can't remove the 'if'. – Alain Collins Nov 09 '15 at 14:56