How to use multiline and json codec at the same time in logstash?

Question

I want to receive log messages in json format. And I also need to merge logs starting with a space ( to send stacktraces in the same email ) with previously found logs.

From the official site documentation, for the first job the required codec is "json". And for the second job, the required codec is "multiline".

How to perform both of the jobs at the same time?

Here is an example log

2014-06-17 14:47:22,490 DEBUG [-] com.tigerit.evr.util.EvrAuthManager (EvrAuthManager.java:61) - User details are good and ready to go

And here is an example stacktrace -

com.bea.core.repackaged.springframework.beans.factory.BeanCreationException: Dependency injection failure: can't find the bean definition about class interface javax.jms.Queue; nested exception is com.bea.core.repackaged.springframework.beans.factory.NoSuchBeanDefinitionException: No unique bean of type [javax.jms.Queue] is defined: No beans of type javax.jms.Queue; owner=com.bea.core.repackaged.springframework.context.support.GenericApplicationContext@1364679d: display name [com.bea.core.repackaged.springframework.context.support.GenericApplicationContext@1364679d]; startup date [Wed Jun 18 10:10:36 BDT 2014]; parent: com.bea.core.repackaged.springframework.context.support.GenericApplicationContext@61932006

        at com.bea.core.repackaged.springframework.jee.inject.Jsr250Metadata.applyInjections(Jsr250Metadata.java:244)

        at com.bea.core.repackaged.springframework.jee.inject.Jsr250Metadata.inject(Jsr250Metadata.java:226)

        at com.bea.core.repackaged.springframework.jee.spi.EjbComponentCreatorBrokerImpl.injection(EjbComponentCreatorBrokerImpl.java:112)

        at com.bea.core.repackaged.springframework.jee.spi.EjbComponentCreatorBrokerImpl.getBean(EjbComponentCreatorBrokerImpl.java:70)

        at weblogic.ejb.container.injection.EjbComponentCreatorImpl.getBean(EjbComponentCreatorImpl.java:68)

        at weblogic.ejb.container.manager.BaseEJBManager.createNewBeanInstance(BaseEJBManager.java:216)

I want to add the lines starting with "at" to merge with the previous log message.

Can you provide some log samples to illustrate what you are asking? — Alcanzar, Jun 17 '14 at 15:53

score 1 · Accepted Answer · answered Jun 18 '14 at 16:14

You can use multiple filters on a given input. For example, you could use the multiline code on the input like in the doc example to merge the stack trace lines:

input {
  stdin {
    codec => multiline {
      pattern => "^\s"
      what => "previous"
    }
  }
}

and then later in your config file you can do something like

filter {
    if [message] =~ /^{.*}$/ {
        json { source => message }
    }
}

So that if you get a line that starts/ends with braces, you can treat that line as json.

How to use multiline and json codec at the same time in logstash?

1 Answers1