Send Tomcat logs to Logstash

Question

I'm trying to implement a centralized log server using ELK stack : Elasticsearch, Logstash, Kibana. It would receive logs from many applications. Basically I have a Tomcat application that uses logback with the following configuration:

<appender name="ROLLING"
    class="ch.qos.logback.core.rolling.RollingFileAppender">
    [...]
</appender>

<appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
    <encoder>
        <pattern>%date %-5level %logger - %msg%n</pattern>
    </encoder>
</appender>


<appender name="stash" class="net.logstash.logback.appender.LogstashTcpSocketAppender">
    <remoteHost>my_remote_host</remoteHost>
    <port>5000</port>

    <encoder class="net.logstash.logback.encoder.LogstashEncoder" />
</appender>


<root level="DEBUG">
    <appender-ref ref="ROLLING" />
    <appender-ref ref="STDOUT" />
    <appender-ref ref="stash" />
</root>

So theoretically this should send via TCP everything the logback logs... On my_remote_host I deployed elasticsearch, logstash and kibana. And they all seem to work. This is the config for logstash:

input {
  stdin {
    type => "human"
  }
  tcp {
    port => 5000
    codec => "json"
    mode => "server"
  }
}

output {
  stdout {}
  elasticsearch {
    host => "my_remote_host"
  }
}

If I type something to stdin in my logstash instance, it successfully indexes my input. Also, if I send by http a request to my_remote_host:5000, it successfully logs the data it receives.

The problem is that logback doesn't seem to send any data using LogstashTcpSocketAppender. Even using a simple SocketAppender, it doesn't work... Am I doing something wrong? It seems to refuse to write to that socket for some reason, but it doesn't complain about anything.

score 1 · Answer 1 · answered May 26 '17 at 22:55

look in catalina.out for logbacks startup logs. also make sure you don't have any firewalls in place (or security groups if you're in aws)

i also use a different codec when sending logs from tomcat / logback to logstach

    tcp {
      port => 4560
      codec => json_lines
      tags => ["app"]
    }

i use this tag in the filter and outputs sections as logback is consuming logs from several places.

score 0 · Answer 2 · answered Feb 16 '17 at 12:26

I am little late with the solution, because I faced with the same issue today. In case if anybody else faces similar problem in the future.

I set the logback <configuration debug="true">

You will notice it is failing to write to OutputStream. So I looked up the versions of the Logback libraries used by the LogstashEncoder, they were older version of 1.1.6 and I was using 1.2.1. I switched to the old version and everything worked fine.

Send Tomcat logs to Logstash

2 Answers2